docparse upload UI

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,13 +1,293 @@
 ---
-title: Docparse
-emoji: ⚡
-colorFrom: blue
-colorTo: blue
+title: docparse
+emoji: 📄
+colorFrom: indigo
+colorTo: gray
 sdk: gradio
-sdk_version: 6.30.0
-python_version: '3.12'
+sdk_version: 5.49.1
+python_version: "3.12"
 app_file: app.py
 pinned: false
+license: mit
+short_description: Extract text, tables and structure from PDF, image, Office
+models:
+- Kedhareswer/docparse
 ---
+# docparse: a very small document parser
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+- **Try it:** [huggingface.co/spaces/Kedhareswer/docparse](https://huggingface.co/spaces/Kedhareswer/docparse) (drop a PDF, image, Word, Excel, CSV or PowerPoint file)
+- **Weights and code:** [huggingface.co/Kedhareswer/docparse](https://huggingface.co/Kedhareswer/docparse): `artifacts/docparse/ocr.pt` (6.0 MB), the `docparse` package, benchmark reports and tests
+- **Source:** [github.com/Kedhareswer/ML-AI](https://github.com/Kedhareswer/ML-AI) (`reasoning_harness/`)
+
+
+`docparse` extracts **everything** from a document (text, headings, paragraphs,
+lists, tables, sheets, formulas, headers/footers, metadata and figure positions)
+and returns it as JSON, Markdown or plain text, with reading order preserved.
+
+| Input | How it is read | Exact? |
+|---|---|---|
+| PDF with a text layer | Words and fonts from the PDF; layout analysis groups them into headings, paragraphs, lists, columns and tables | Text is exact |
+| Scanned PDF pages (no text layer) | Rendered at 200 dpi, then the OCR pipeline below | OCR |
+| Images: PNG, JPG, TIFF (multi-page), BMP, WEBP, GIF | OCR pipeline | OCR |
+| Word `.docx` | Document structure: styles → headings/lists, tables (merged cells), text boxes, headers/footers, properties | Exact |
+| Excel `.xlsx` | Every sheet: values, formulas, merged ranges, dates, separate tables split on empty rows/columns | Exact |
+| CSV / TSV | Delimiter and encoding sniffed | Exact |
+| `.doc`, `.rtf`, `.odt`, `.xls`, `.ods`, `.ppt`, `.pptx`, `.odp` | Converted with LibreOffice (if installed), then parsed as above | Exact |
+
+The design choice: **a model is only used where one is needed.** Born-digital
+files already contain their text; reading their structure is exact and much
+faster than any network. The neural network reads pixels: images, scans and
+(optionally, `--ocr-images`) pictures embedded inside documents.
+
+## The model
+
+A CRNN line recognizer trained **from scratch** (no pretrained weights) on a
+4-core CPU:
+
+- **1,484,357 parameters, 6.0 MB** (`artifacts/docparse/ocr.pt`): 6 convolution
+  layers, a 2-layer bidirectional LSTM and CTC decoding.
+- Input: a grayscale text line, 32 px high, any width. Output: the text plus a
+  confidence (mean probability of the emitted characters).
+- Characters: printable ASCII plus `• € £ ° ©`. Typographic quotes, dashes and
+  accented letters are folded to ASCII (`é` → `e`).
+- Training: 480,000 synthetic lines (4 rounds of 120,000 freshly rendered
+  lines), 27 fonts (DejaVu, Liberation, FreeFont, Inter, JetBrains Mono),
+  text from app reviews and medical abstracts in this repository, plus
+  generated numbers, prices, dates, emails, URLs, IDs and list items. Lines are
+  degraded like scans and screenshots: blur, low resolution, noise, JPEG,
+  rotation, uneven light, inverted colours and table-rule remnants. Training
+  took 80 minutes.
+
+Everything around the network is classical image processing (numpy only):
+adaptive binarization (dark-on-light and light-on-dark), speck removal,
+deskew, ruled-table detection from long horizontal/vertical strokes, frame
+and outline removal, recursive XY-cut for columns and lines, then grouping
+lines into headings (by x-height), paragraphs (by line spacing), list items,
+and borderless tables (rows whose gaps line up).
+
+## Results
+
+All numbers are from `docparse bench` (`benchmarks/docparse/report.json`).
+CER = character error rate (edit distance / reference length).
+
+**1. Single lines.** Test fonts **Carlito and Caladea were never used in
+training** (they are metric clones of Calibri and Cambria, the Office
+defaults), and the test sentences come from a separate corpus.
+
+| Test set | Lines | CER | Word error | Lines exactly right |
+|---|---:|---:|---:|---:|
+| Held-out fonts, clean | 2,000 | **0.27%** | 2.3% | 93.3% |
+| Held-out fonts, degraded | 2,000 | **0.93%** | 6.0% | 84.8% |
+| Training fonts, unseen text, degraded | 2,000 | 0.24% | 1.2% | 95.6% |
+| Untrained network (same architecture), degraded | 500 | 99.7% | 99.8% | 0% |
+
+About 3 ms per line on CPU.
+
+**2. Synthetic A4 pages** (held-out fonts and text; titles, section headings,
+paragraphs, bullet lists, ruled tables, key/value tables, two-column sections).
+"Scanned" pages are rotated up to ±2°, blurred, noised and JPEG-compressed.
+
+| | Clean (40 pages) | Scanned (40 pages) |
+|---|---:|---:|
+| Page CER, in reading order (mean / median) | 0.32% / 0.00% | 1.65% / 0.19% |
+| Word F1 (order-free) | 98.9% | 97.7% |
+| Headings found | 86.7% | 85.7% |
+| Tables found with the right rows × columns | 100% | 100% |
+| Table cells exactly right | 98.5% | 90.2% |
+| Seconds per page (CPU) | 0.9 | 1.3 |
+
+A few hard pages pull the scanned mean up; the median page is almost perfect.
+Missed headings are mostly short numbered section titles ("1. Results") read
+as list items or paragraphs: headings are detected by letter size, not by
+boldness.
+
+**3. Real pages from this repository.** 40 pages of the three PDFs in this
+repo (two Jupyter notebook exports and a slide deck) were rendered to
+150 dpi images and read **from pixels only**. Each PDF's own text layer is the
+reference.
+
+| PDF | Pages | Median CER (reading order) | Median word F1 |
+|---|---:|---:|---:|
+| Customer Behavior Analysis (notebook) | 15 | 0.7% | 92.5% |
+| Credit Card Fraud Detection (notebook) | 11 | 2.3% | 94.8% |
+| PPT.pdf (slides with graphics) | 14 | 32.3% | 82.5% |
+| **All** | **40** | — | **91.9%** (mean 86.3%) |
+
+Most slide-deck error is reading order: slides scatter text boxes, and the
+PDF's own order differs from top-to-bottom reading. Text inside charts (axis
+labels, tick numbers) is also read as text. The weakest notebook pages are
+mostly plots with a few short labels.
+
+## ParseBench (LlamaIndex)
+
+[ParseBench](https://huggingface.co/datasets/llamaindex/ParseBench)
+([paper](https://arxiv.org/abs/2604.08538), [code](https://github.com/run-llama/ParseBench))
+is LlamaIndex's public benchmark for document parsers: **2,078 real enterprise
+pages** (insurance filings, financial reports, government documents, papers),
+1,211 documents and 169,011 human-verified test rules in five dimensions.
+Scoring is deterministic rule matching, with no LLM judge.
+
+docparse was run on the **full** dataset with the official harness
+(parse-bench 1.0.4) on a 4-core CPU: all 2,078 pages, 0 failures, median
+0.5 s per page. The comparison rows are the "Open Source - Local" entries of
+ParseBench's published leaderboard.
+
+| Parser | Overall | Tables | Charts | Content faithfulness | Semantic formatting | Visual grounding |
+|---|---:|---:|---:|---:|---:|---:|
+| PyMuPDF4LLM | 53.5 | 72.0 | 2.2 | 79.3 | 52.0 | 61.9 |
+| Warp Ingest | 40.2 | 57.5 | 7.0 | 70.8 | 45.8 | 19.8 |
+| LiteParse (no OCR) | 36.9 | 42.4 | 1.8 | 70.0 | 38.5 | 31.8 |
+| **docparse** | **36.2** | **42.1** | **1.0** | **72.5** | **35.5** | **30.0** |
+| OpenDataLoader | 29.4 | 35.2 | 0.9 | 66.1 | 34.1 | 10.8 |
+| pdf-inspector | 26.6 | 26.6 | 5.3 | 56.1 | 35.1 | 9.9 |
+| MarkItDown | 18.6 | 15.8 | 2.0 | 64.5 | 0.9 | 9.9 |
+| PyMuPDF (text) | 16.0 | 0.0 | 0.0 | 68.3 | 1.0 | 10.9 |
+| pypdf | 14.9 | 0.0 | 0.0 | 62.5 | 0.9 | 10.9 |
+
+Overall is the mean of the five dimensions, as on the official leaderboard.
+For scale: the top of the full leaderboard (cloud services and multi-billion
+parameter vision-language models) reaches 75–90.
+
+**What this says honestly**
+
+- docparse is mid-table among local open-source parsers: ahead of pypdf,
+  PyMuPDF, MarkItDown, pdf-inspector and OpenDataLoader; just behind LiteParse
+  (36.9); behind Warp Ingest and clearly behind PyMuPDF4LLM.
+- Its **content faithfulness (72.5)** is second among local parsers. It is
+  strongest on simple (89.0) and multi-column (76.5) pages, and weakest on
+  handwriting (27.4), scans (56.1) and non-English text (58.8). That matches the
+  OCR model's limits: printed Latin text only.
+- **Charts (1.0):** ParseBench wants each chart's data points as a table.
+  docparse finds charts and keeps their labels but does not convert them to
+  data. No local parser scores above 7.
+- **Tables (42.1)** are limited by pdfplumber's ruled-grid detection on
+  borderless financial tables; 32.3 on "hard" tables.
+- **Formatting (35.5):** bold 29%, italic 23%, superscript 59%, subscript 57%
+  of rules pass. Underline, strikeout, highlight, LaTeX and code blocks are not
+  produced (0%).
+- **Visual grounding (30.0):** paragraph boxes and reading order are decent;
+  figure regions and box granularity often differ from the annotation.
+
+**What the benchmark changed in docparse.** The first full run scored 33.66; the final run scored 36.2.
+Its failures led to fixes that are now part of docparse itself:
+
+| Change | Effect |
+|---|---|
+| Bold/italic from PDF font names, `<sup>`/`<sub>` from glyph size and baseline | Semantic formatting 24.0 → 35.5 |
+| Running header/footer and printed page-number detection | Header rules 0 → 9.9%, footer rules 0 → 12.2%; content faithfulness 71.8 → 72.5 |
+| Rotated margin text (e.g. arXiv stamps) kept out of the reading flow | It no longer turns body text into a fake table |
+| Vector charts and diagrams detected as figures, their labels attached | Visual grounding 28.9 → 30.0 |
+| Rows stacked in one cell of rule-less tables split into records | Correct rows in insurance rate tables |
+| A malformed PDF link annotation no longer fails the whole page | 9 failing pages → 0 |
+
+Reproduce (parse-bench needs Python 3.12+):
+
+```bash
+pip install "parse-bench[fast]"
+python scripts/parsebench_docparse.py download          # ~600 MB
+python scripts/parsebench_docparse.py run docparse --max_concurrent 3
+python scripts/parsebench_summary.py output/docparse leaderboard.csv
+```
+
+`docparse/parsebench.py` registers the pipeline through ParseBench's public
+extension API. Each page is sent as Markdown with HTML tables, page
+header/footer fields, and layout boxes in ParseBench's canonical classes.
+Aggregate results are in `benchmarks/parsebench/report.json`.
+
+## ExtractBench (LlamaIndex): small local check
+
+[ExtractBench](https://github.com/run-llama/ExtractBench) scores **schema-guided
+extraction**: a document plus a JSON Schema in, schema-valid JSON plus evidence
+boxes out. A parser cannot do this alone; an LLM fills the schema. So docparse
+was tested as the **parse stage** of a two-stage pipeline (as ExtractBench's own
+parse → LLM baselines are), against plain pypdf text, with everything else fixed:
+the same model (Qwen2.5-3B-Instruct, 4-bit, llama.cpp on 4 CPU cores), the same
+prompt and 12,288-token context budget, and the official scorer.
+
+| Parse stage (6-document test set) | Value F1 | Precision | Recall |
+|---|---:|---:|---:|
+| **docparse** | **35.8** | 63.8 | 32.5 |
+| pypdf | 32.9 | 66.3 | 29.8 |
+
+docparse helped most on a scanned document (Value F1 19.6 vs 8.0, from OCR) and
+an earnings deck with tables (35.0 vs 25.0). It lost on the 66-page document
+(1.4 vs 5.2), where both arms could send only the first few pages.
+
+**Read this carefully.** This is 6 documents, so a 2.9-point gap is not
+statistically meaningful. Absolute scores are far below the leaderboard (top
+systems score ~96), because a 3B model with a 12k context must cut long documents
+(up to 55 of 66 pages dropped) and sometimes even shorten the schema. Grounding is
+0 for both arms because this pipeline does not return evidence boxes yet. A real
+ExtractBench number needs a strong model over all 370 documents (ExtractBench
+estimates $10 to $1,677 per full run, depending on the model). Per-document
+details are in `benchmarks/extractbench/report.json`.
+
+Reproduce: start an OpenAI-compatible server (`python -m llama_cpp.server --model
+qwen2.5-3b-instruct-q4_k_m.gguf --n_ctx 12288 --port 8088`), then
+`DOCPARSE_LLM_URL=http://127.0.0.1:8088 python scripts/extractbench_docparse.py run
+docparse_extract_local --test` (and `pypdf_extract_local`).
+
+**Structure parsers** (PDF text layer, DOCX, XLSX, CSV) are tested for exact
+output in `tests/test_docparse.py`.
+
+## Use it
+
+```bash
+pip install -e '.[docs]'          # torch, numpy, pillow, pdfplumber, pypdfium2, python-docx, openpyxl
+
+docparse extract report.pdf                      # Markdown to stdout
+docparse extract scan.png --format json -o scan.json
+docparse extract invoice.pdf --ocr always        # OCR every page, even with a text layer
+docparse extract deck.docx --ocr-images          # also read text inside embedded pictures
+docparse extract *.xlsx --format md -o out/      # many files -> a directory
+docparse ui                                      # browser UI on http://127.0.0.1:7862
+```
+
+```python
+from docparse import extract
+from docparse.extract import to_markdown
+
+doc = extract("statement.pdf")          # dict: source, format, metadata, pages, blocks, sheets, warnings
+for b in doc["blocks"]:
+    print(b["type"], b.get("page"), b.get("text") or b.get("rows"))
+print(to_markdown(doc))
+```
+
+Each block has a `type` (`heading`, `paragraph`, `list_item`, `table`,
+`figure`, `header`, `footer`), a `source` (`text-layer`, `structure` or
+`ocr`), and where meaningful a `page`, `bbox`, `level`, `sheet`, `range` and
+an OCR `confidence`. Excel sheets also list formulas, merged ranges, charts and
+images under `sheets`.
+
+## Train or fine-tune your own
+
+```bash
+python scripts/build_doc_corpus.py                        # rebuild the text corpora
+docparse train-ocr --run runs/ocr --rounds 4              # ~80 min on 4 CPU cores
+docparse train-ocr --run runs/ocr --rounds 6 --resume     # continue / extend training
+docparse bench --model runs/ocr/ocr.pt --output runs/ocr-bench
+```
+
+To specialise it (your fonts, your vocabulary, e.g. invoices in one
+typeface), add font files to `TRAIN_FONTS` and text to `docparse/data/corpus_train.txt`
+in `docparse/ocr/render.py`, then train or resume. A checkpoint is written
+after every round.
+
+## Limits
+
+- **Printed text only.** No handwriting. Latin script only (ASCII plus a few
+  symbols); accents are dropped and other scripts are not read.
+- Tables: ruled grids and borderless tables with aligned columns are found.
+  Cells that span several rows or columns are kept in the first cell. Very
+  irregular tables may come out as text.
+- Reading order is top-to-bottom, columns left to right. Magazine layouts,
+  slides with scattered boxes, sidebars and text running around pictures can
+  be read out of order.
+- Pictures are reported as `figure` blocks with a position. Charts are not
+  converted to data.
+- Photos of documents taken at an angle (perspective), curved pages and
+  very low resolution (text under about 10 px high) are not handled well.
+- Born-digital PDFs are only as good as their text layer: some PDFs embed
+  fonts without a Unicode mapping. Use `--ocr always` for those.
+- Besides ParseBench, the benchmarks are synthetic pages and three real PDFs from
+  this repository. No direct comparison with Tesseract was run here.

app.py

@@ -0,0 +1,22 @@
+# Entry point for the free Gradio-SDK Space: Spaces run `python app.py` and route
+# traffic to port 7860, so this starts docparse's own upload UI there.
+import os
+
+os.environ.setdefault("OMP_NUM_THREADS", "2")
+
+try:  # ZeroGPU (the only free Gradio hardware) refuses to start without one
+    import spaces
+
+    @spaces.GPU
+    def _unused():
+        pass
+
+    # spaces normally reports this from gr.Blocks.launch, which this app never calls
+    from spaces.zero import client
+    client.startup_report()
+except ImportError:
+    pass
+
+from docparse.ui import serve
+
+serve(port=7860, host="0.0.0.0", model="artifacts/docparse/ocr.pt")

artifacts/docparse/manifest.json

@@ -0,0 +1,14 @@
+{
+  "model": "LineOCR (CRNN: 6 conv layers + 2-layer BiLSTM, CTC)",
+  "parameters": 1484357,
+  "weights_bytes": 5958917,
+  "sha256": "ed073bd91ee8d12727b335ee08154e98c2fcc35c54e3083b7c25dd49cff1da8d",
+  "input": "grayscale text line, height 32, any width",
+  "charset": "printable ASCII + \u2022 \u20ac \u00a3 \u00b0 \u00a9",
+  "training_data": "480,000 synthetic lines (4 rounds x 120,000, fresh seeds each round), 27 training fonts",
+  "held_out": "Carlito and Caladea font families (metric clones of Calibri/Cambria) and a separate text corpus",
+  "train_seconds": 4829.3,
+  "device": "cpu (4 cores)",
+  "torch": "2.6.0+cpu",
+  "initialisation": "random (torch.manual_seed 0); no pretrained weights"
+}

artifacts/docparse/ocr.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed073bd91ee8d12727b335ee08154e98c2fcc35c54e3083b7c25dd49cff1da8d
+size 5958917

artifacts/docparse/train.log

@@ -0,0 +1,44 @@
+round 1 step 0/2500 loss 12.8047 lr 8.00e-05 elapsed 100s
+round 1 step 250/2500 loss 5.2012 lr 5.11e-04 elapsed 219s
+round 1 step 500/2500 loss 3.2770 lr 1.41e-03 elapsed 357s
+round 1 step 750/2500 loss 0.8169 lr 1.98e-03 elapsed 536s
+round 1 step 1000/2500 loss 0.2078 lr 2.00e-03 elapsed 657s
+round 1 step 1250/2500 loss 0.0972 lr 1.99e-03 elapsed 782s
+round 1 step 1500/2500 loss 0.0859 lr 1.97e-03 elapsed 892s
+round 1 step 1750/2500 loss 0.0785 lr 1.95e-03 elapsed 1010s
+round 1 step 2000/2500 loss 0.0683 lr 1.92e-03 elapsed 1112s
+round 1 step 2250/2500 loss 0.0497 lr 1.88e-03 elapsed 1211s
+{"round": 1, "train_loss": 0.0498, "val_train_fonts": {"lines": 3000, "cer": 0.01403, "wer": 0.06834, "line_exact": 0.8077}, "test_heldout_fonts": {"lines": 3000, "cer": 0.02851, "wer": 0.13867, "line_exact": 0.6483}, "elapsed_s": 1325.6}
+round 2 step 0/2500 loss 0.0188 lr 1.84e-03 elapsed 1463s
+round 2 step 250/2500 loss 0.0467 lr 1.79e-03 elapsed 1687s
+round 2 step 500/2500 loss 0.0386 lr 1.73e-03 elapsed 1922s
+round 2 step 750/2500 loss 0.0529 lr 1.67e-03 elapsed 2126s
+round 2 step 1000/2500 loss 0.0415 lr 1.60e-03 elapsed 2214s
+round 2 step 1250/2500 loss 0.0349 lr 1.53e-03 elapsed 2311s
+round 2 step 1500/2500 loss 0.0321 lr 1.46e-03 elapsed 2396s
+round 2 step 1750/2500 loss 0.0271 lr 1.38e-03 elapsed 2494s
+round 2 step 2000/2500 loss 0.0233 lr 1.30e-03 elapsed 2587s
+round 2 step 2250/2500 loss 0.0313 lr 1.22e-03 elapsed 2679s
+{"round": 2, "train_loss": 0.028, "val_train_fonts": {"lines": 3000, "cer": 0.0051, "wer": 0.02588, "line_exact": 0.916}, "test_heldout_fonts": {"lines": 3000, "cer": 0.01489, "wer": 0.08693, "line_exact": 0.779}, "elapsed_s": 2781.7}
+round 3 step 0/2500 loss 0.0044 lr 1.14e-03 elapsed 2882s
+round 3 step 250/2500 loss 0.0210 lr 1.05e-03 elapsed 2970s
+round 3 step 500/2500 loss 0.0229 lr 9.65e-04 elapsed 3064s
+round 3 step 750/2500 loss 0.0220 lr 8.80e-04 elapsed 3154s
+round 3 step 1000/2500 loss 0.0232 lr 7.96e-04 elapsed 3246s
+round 3 step 1250/2500 loss 0.0204 lr 7.13e-04 elapsed 3338s
+round 3 step 1500/2500 loss 0.0172 lr 6.33e-04 elapsed 3433s
+round 3 step 1750/2500 loss 0.0147 lr 5.55e-04 elapsed 3523s
+round 3 step 2000/2500 loss 0.0155 lr 4.80e-04 elapsed 3613s
+round 3 step 2250/2500 loss 0.0136 lr 4.09e-04 elapsed 3695s
+{"round": 3, "train_loss": 0.0135, "val_train_fonts": {"lines": 3000, "cer": 0.00235, "wer": 0.01228, "line_exact": 0.958}, "test_heldout_fonts": {"lines": 3000, "cer": 0.00977, "wer": 0.06076, "line_exact": 0.834}, "elapsed_s": 3805.6}
+round 4 step 0/2500 loss 0.0053 lr 3.42e-04 elapsed 3904s
+round 4 step 250/2500 loss 0.0118 lr 2.80e-04 elapsed 3994s
+round 4 step 500/2500 loss 0.0119 lr 2.24e-04 elapsed 4084s
+round 4 step 750/2500 loss 0.0112 lr 1.73e-04 elapsed 4172s
+round 4 step 1000/2500 loss 0.0106 lr 1.28e-04 elapsed 4261s
+round 4 step 1250/2500 loss 0.0099 lr 8.94e-05 elapsed 4353s
+round 4 step 1500/2500 loss 0.0120 lr 5.75e-05 elapsed 4436s
+round 4 step 1750/2500 loss 0.0094 lr 3.25e-05 elapsed 4529s
+round 4 step 2000/2500 loss 0.0099 lr 1.44e-05 elapsed 4627s
+round 4 step 2250/2500 loss 0.0097 lr 3.59e-06 elapsed 4717s
+{"round": 4, "train_loss": 0.009, "val_train_fonts": {"lines": 3000, "cer": 0.00168, "wer": 0.00907, "line_exact": 0.9683}, "test_heldout_fonts": {"lines": 3000, "cer": 0.00848, "wer": 0.05398, "line_exact": 0.8533}, "elapsed_s": 4829.3}

artifacts/docparse/train_info.json

@@ -0,0 +1,77 @@
+{
+  "parameters": 1484357,
+  "rounds": 4,
+  "lines_per_round": 120000,
+  "batch_size": 48,
+  "seed": 0,
+  "history": [
+    {
+      "round": 1,
+      "train_loss": 0.0498,
+      "val_train_fonts": {
+        "lines": 3000,
+        "cer": 0.01403,
+        "wer": 0.06834,
+        "line_exact": 0.8077
+      },
+      "test_heldout_fonts": {
+        "lines": 3000,
+        "cer": 0.02851,
+        "wer": 0.13867,
+        "line_exact": 0.6483
+      },
+      "elapsed_s": 1325.6
+    },
+    {
+      "round": 2,
+      "train_loss": 0.028,
+      "val_train_fonts": {
+        "lines": 3000,
+        "cer": 0.0051,
+        "wer": 0.02588,
+        "line_exact": 0.916
+      },
+      "test_heldout_fonts": {
+        "lines": 3000,
+        "cer": 0.01489,
+        "wer": 0.08693,
+        "line_exact": 0.779
+      },
+      "elapsed_s": 2781.7
+    },
+    {
+      "round": 3,
+      "train_loss": 0.0135,
+      "val_train_fonts": {
+        "lines": 3000,
+        "cer": 0.00235,
+        "wer": 0.01228,
+        "line_exact": 0.958
+      },
+      "test_heldout_fonts": {
+        "lines": 3000,
+        "cer": 0.00977,
+        "wer": 0.06076,
+        "line_exact": 0.834
+      },
+      "elapsed_s": 3805.6
+    },
+    {
+      "round": 4,
+      "train_loss": 0.009,
+      "val_train_fonts": {
+        "lines": 3000,
+        "cer": 0.00168,
+        "wer": 0.00907,
+        "line_exact": 0.9683
+      },
+      "test_heldout_fonts": {
+        "lines": 3000,
+        "cer": 0.00848,
+        "wer": 0.05398,
+        "line_exact": 0.8533
+      },
+      "elapsed_s": 4829.3
+    }
+  ]
+}

docparse/__init__.py

@@ -0,0 +1,10 @@
+"""docparse: extract text, tables and structure from PDF, images, Word and Excel files.
+
+Born-digital formats are parsed exactly from their own structure. Images and scanned
+PDF pages go through a small OCR network trained from scratch (docparse.ocr).
+"""
+
+
+def extract(path, **kw):
+    from .extract import extract as _extract
+    return _extract(path, **kw)

docparse/bench.py

@@ -0,0 +1,165 @@
+"""Benchmarks: line OCR, synthetic pages, and real PDF pages read from pixels.
+
+1. Lines: held-out fonts (Carlito/Caladea) + held-out text, clean and degraded; the
+   training fonts on unseen text are reported separately, as is an untrained network.
+2. Pages: synthetic A4 pages (held-out fonts/text), clean renders and simulated scans.
+   Scores: page CER in reading order, order-free word F1, heading recall, table recovery.
+3. Real pages: born-digital PDFs from this repository rendered to images and OCR'd; the
+   PDF's own text layer is the reference. Reading order of complex pages differs, so
+   the order-free word F1 is the fairer score there.
+"""
+import json
+import random
+import time
+from collections import Counter
+from pathlib import Path
+
+import numpy as np
+import torch
+
+from .ocr.metrics import cer, summary
+from .ocr.model import LineOCR, Recognizer
+from .ocr.render import TRAIN_FONTS, available, dataset, render_line, sample_text
+from .synth_pages import make_page, truth_text
+
+ROOT = Path(__file__).resolve().parents[1]
+REAL_PDFS = ["../Credit Card/Credit Card Fraud Detection.pdf", "../ML/PPT.pdf",
+             "../Customer Analysis/vertopal.com_Customer_Behavior_Analysis.pdf"]
+
+
+def _norm(t):
+    return " ".join(str(t).split())
+
+
+def word_f1(pred, ref):
+    p, r = Counter(pred.split()), Counter(ref.split())
+    hit = sum((p & r).values())
+    if not hit:
+        return 0.0
+    prec, rec = hit / sum(p.values()), hit / sum(r.values())
+    return 2 * prec * rec / (prec + rec)
+
+
+def lines(rec, count=2000):
+    out = {}
+    for name, split, degrade, seed in [("heldout_fonts_clean", "test", False, 7_000_001),
+                                       ("heldout_fonts_degraded", "test", True, 7_000_002),
+                                       ("train_fonts_unseen_text_degraded", "train", True, 7_000_003)]:
+        if split == "train":  # training fonts with held-out text, so no sentence was seen in training
+            ims, txt = [], []
+            fonts = available(TRAIN_FONTS)
+            for i in range(count):
+                r = random.Random(seed * 1_000_003 + i)
+                txt.append(sample_text(r, "test"))
+                ims.append(render_line(txt[-1], r.choice(fonts), r.randint(14, 44), r, degrade))
+        else:
+            ims, txt = dataset(count, seed, split, degrade)
+        t0 = time.time()
+        preds = [p for p, _ in rec.read(ims, batch=64)]
+        res = summary(preds, txt)
+        res["ms_per_line"] = round(1000 * (time.time() - t0) / count, 2)
+        res["examples"] = [{"truth": t, "pred": p} for t, p in list(zip(txt, preds))[:8]]
+        out[name] = res
+    # Untrained control: same architecture, random weights.
+    torch.manual_seed(0)
+    blank = Recognizer.__new__(Recognizer)
+    blank.net, blank.device = LineOCR().eval(), "cpu"
+    ims, txt = dataset(500, 7_000_002, "test", True)
+    out["untrained_network_heldout_degraded"] = summary([p for p, _ in blank.read(ims)], txt)
+    return out
+
+
+def _flat(blocks):
+    out = []
+    for b in blocks:
+        if b["type"] == "table":
+            out += [" ".join(str(c) for c in row if str(c)) for row in b["rows"]]
+        elif b.get("text"):
+            out.append(b["text"])
+    return _norm(" ".join(out))
+
+
+def pages(rec, count=40):
+    from .ocr.page import analyze
+    res = {}
+    for scan in (False, True):
+        cers, f1s, heads, head_hits, tabs, tab_hits, cells, cell_hits, secs = [], [], 0, 0, 0, 0, 0, 0, []
+        for i in range(count):
+            img, truth = make_page(8_000_000 + i + (100_000 if scan else 0), "test", scan)
+            t0 = time.time()
+            blocks, _ = analyze(img, rec)
+            secs.append(time.time() - t0)
+            ref, pred = _norm(truth_text(truth)), _flat(blocks)
+            cers.append(cer(pred, ref))
+            f1s.append(word_f1(pred, ref))
+            pheads = [b["text"] for b in blocks if b["type"] == "heading"]
+            for b in truth:
+                if b["type"] == "heading":
+                    heads += 1
+                    head_hits += any(cer(p, b["text"]) < 0.2 for p in pheads)
+                if b["type"] == "table":
+                    tabs += 1
+                    shape = (len(b["rows"]), len(b["rows"][0]))
+                    match = [p for p in blocks if p["type"] == "table" and (len(p["rows"]), len(p["rows"][0])) == shape]
+                    if match:
+                        tab_hits += 1
+                        best = max(match, key=lambda p: sum(_norm(x) == _norm(y) for rp, rt in zip(p["rows"], b["rows"])
+                                                            for x, y in zip(rp, rt)))
+                        cell_hits += sum(_norm(x) == _norm(y) for rp, rt in zip(best["rows"], b["rows"]) for x, y in zip(rp, rt))
+                    cells += shape[0] * shape[1]
+        res["scanned" if scan else "clean"] = {
+            "pages": count, "page_cer_reading_order": round(float(np.mean(cers)), 4),
+            "page_cer_median": round(float(np.median(cers)), 4), "word_f1": round(float(np.mean(f1s)), 4),
+            "heading_recall": round(head_hits / max(1, heads), 4), "headings": heads,
+            "tables_found_with_correct_shape": round(tab_hits / max(1, tabs), 4), "tables": tabs,
+            "table_cells_exact": round(cell_hits / max(1, cells), 4), "cells": cells,
+            "seconds_per_page": round(float(np.mean(secs)), 2)}
+    return res
+
+
+def real_pdfs(rec, dpi=150, max_pages=40):
+    """Render born-digital pages to pixels, OCR them, and compare with the PDF's text layer."""
+    import pdfplumber
+    import pypdfium2 as pdfium
+    from .ocr.page import analyze
+    rows = []
+    for rel in REAL_PDFS:
+        path = (ROOT / rel).resolve()
+        if not path.exists():
+            continue
+        pdf_ = pdfium.PdfDocument(str(path))
+        with pdfplumber.open(path) as pl:
+            for n, page in enumerate(pl.pages):
+                ref = _norm(page.extract_text() or "")
+                if len(ref) < 50 or len(rows) >= max_pages:
+                    continue
+                img = pdf_[n].render(scale=dpi / 72).to_pil()
+                t0 = time.time()
+                blocks, _ = analyze(img, rec)
+                pred = _flat(blocks)
+                rows.append({"file": path.name, "page": n + 1, "ref_chars": len(ref),
+                             "cer_reading_order": round(cer(pred, ref), 4), "word_f1": round(word_f1(pred, ref), 4),
+                             "seconds": round(time.time() - t0, 2)})
+        pdf_.close()
+    agg = {"pages": len(rows), "dpi": dpi,
+           "mean_word_f1": round(float(np.mean([r["word_f1"] for r in rows])), 4) if rows else None,
+           "median_word_f1": round(float(np.median([r["word_f1"] for r in rows])), 4) if rows else None,
+           "mean_cer_reading_order": round(float(np.mean([r["cer_reading_order"] for r in rows])), 4) if rows else None}
+    return {"summary": agg, "pages": rows}
+
+
+def run(model, output, line_count=2000, page_count=40, real_pages=40):
+    torch.set_num_threads(4)
+    out = Path(output)
+    out.mkdir(parents=True, exist_ok=True)
+    rec = Recognizer(str(model))
+    report = {"model": str(model), "parameters": sum(p.numel() for p in rec.net.parameters()),
+              "weights_mb": round(Path(model).stat().st_size / 2 ** 20, 2)}
+    report["lines"] = lines(rec, line_count)
+    print(json.dumps({k: {m: v for m, v in r.items() if m != "examples"} for k, r in report["lines"].items()}, indent=1))
+    report["pages"] = pages(rec, page_count)
+    print(json.dumps(report["pages"], indent=1))
+    report["real_pdf_pages"] = real_pdfs(rec, max_pages=real_pages)
+    print(json.dumps(report["real_pdf_pages"]["summary"], indent=1))
+    (out / "report.json").write_text(json.dumps(report, indent=2))
+    return report

docparse/cli.py

@@ -0,0 +1,78 @@
+"""Command line: docparse extract | train-ocr | bench | ui."""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="docparse")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    e = sub.add_parser("extract", help="extract text, tables and structure from a document")
+    e.add_argument("files", nargs="+", help="PDF, image, DOCX/DOC, XLSX/XLS, CSV, PPTX ...")
+    e.add_argument("--format", default="md", choices=["md", "json", "text"])
+    e.add_argument("--ocr", default="auto", choices=["auto", "always", "never"],
+                   help="auto: OCR only PDF pages that have no text layer")
+    e.add_argument("--ocr-images", action="store_true", help="also read text inside pictures embedded in documents")
+    e.add_argument("--model", help="OCR weights (default artifacts/docparse/ocr.pt)")
+    e.add_argument("--dpi", type=int, default=200, help="render resolution for scanned PDF pages")
+    e.add_argument("--password", help="password for encrypted PDFs")
+    e.add_argument("-o", "--output", help="output file (one input) or directory (several inputs)")
+
+    t = sub.add_parser("train-ocr", help="train the line recognizer from scratch on synthetic lines")
+    t.add_argument("--run", required=True)
+    t.add_argument("--rounds", type=int, default=4)
+    t.add_argument("--lines-per-round", type=int, default=120_000)
+    t.add_argument("--batch-size", type=int, default=48)
+    t.add_argument("--lr", type=float, default=2e-3)
+    t.add_argument("--seed", type=int, default=0)
+    t.add_argument("--threads", type=int, default=4)
+    t.add_argument("--resume", action="store_true")
+
+    b = sub.add_parser("bench", help="benchmark OCR on held-out lines, synthetic pages and real PDF pages")
+    b.add_argument("--model", default="artifacts/docparse/ocr.pt")
+    b.add_argument("--output", required=True)
+    b.add_argument("--lines", type=int, default=2000)
+    b.add_argument("--pages", type=int, default=40)
+    b.add_argument("--real-pages", type=int, default=40)
+
+    u = sub.add_parser("ui", help="local browser UI: drop a file, get Markdown and JSON")
+    u.add_argument("--port", type=int, default=7862)
+    u.add_argument("--host", default="127.0.0.1")
+    u.add_argument("--model")
+
+    args = ap.parse_args(argv)
+    if args.cmd == "extract":
+        from .extract import extract, to_markdown, to_text
+        for f in args.files:
+            doc = extract(f, ocr=args.ocr, ocr_images=args.ocr_images, model=args.model, dpi=args.dpi,
+                          password=args.password)
+            body = (json.dumps(doc, indent=2, ensure_ascii=False, default=str) if args.format == "json"
+                    else to_markdown(doc) if args.format == "md" else to_text(doc) + "\n")
+            if args.output and len(args.files) > 1:
+                out = Path(args.output)
+                out.mkdir(parents=True, exist_ok=True)
+                (out / (Path(f).name + "." + args.format)).write_text(body, encoding="utf-8")
+            elif args.output:
+                Path(args.output).write_text(body, encoding="utf-8")
+            else:
+                sys.stdout.write(body)
+            for w in doc["warnings"]:
+                print(f"warning: {Path(f).name}: {w}", file=sys.stderr)
+    elif args.cmd == "train-ocr":
+        from .ocr.train import train
+        _, info = train(args.run, args.rounds, args.lines_per_round, args.batch_size, args.lr, args.seed,
+                        args.threads, args.resume, log=lambda m: print(m, flush=True))
+        print(json.dumps(info["history"][-1], indent=2))
+    elif args.cmd == "bench":
+        from .bench import run
+        run(args.model, args.output, args.lines, args.pages, args.real_pages)
+    elif args.cmd == "ui":
+        from .ui import serve
+        serve(args.port, args.host, args.model)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docparse/data/corpus_test.txt

@@ -0,0 +1,325 @@
+Type C effects (those effects related to an increased frequency of 'spontaneous ' disease) are difficult to study, however, and continue to pose a pharmacoepidemiological challenge
+DESIGN : Case-control association testing
+The app/ website is difficult to use and the UI is as good as a brick. Windows 98 is easier to use. You can't contact customer support via the phone, at all. If you can't log in you can't get to their email to contact them and when you finally hear back it's "violated terms of service" and will not elaborate. IV read the TOS and there's nothing I'm doing wrong. Can't use the app/website on multiple devices or it will lose its ****. Relies on community mangers to help solve claims but it dosent help at all. I had to google a phone number and email to put in a support ticket while being locked out of my account. The phone goes to an automated voice that only tells you, to refer to their help tab that has useless information. LinkedIn needs competition because this is a huge pain to deal with. I strongly suggest you look for an alternative to LinkedIn.
+Comparison of the frequency of lymphocyte expression of PIBF in pregnant versus non-pregnant women were made
+These results clearly show that the short form acts as a dominant negative inhibitor through the formation of inactive heterodimers, resulting in an inhibition of Janus kinase 2 (JAK2) activation
+Very few autonomic tests have been done on pediatric patients thus far
+PARTICIPANTS : The 562 participating adolescents represented 92% of students in mandatory classes
+Both current and previous prescribing for sore throat increase reattendance
+D19S574 showed a significant effect (P=.015) in Caucasian patients with MS due to the increased frequency of a single allele (P=.002)
+A key feature of the method is isolation and reverse transcription of mRNA followed by PCR amplification of lysosomal alpha-glucosidase cDNA with M13-extended primers
+It is concluded that ENS is more effective for immediate relief of myofascial trigger point pain than EMS, and EMS has a better effect on immediate release of muscle tightness than ENS
+An increase in the SDS concentration reduces the number of different species present in micelles
+These levels were determined in patients undergoing three types of therapy : non-in vitro fertilization (IVF), IVF-embryo transfer (ET), and frozen ET
+In all complete honesty, LinkedIn as a great app and it's net work is good, but seriously, the amount of emails that you get from this company probably emits as much greenhouse gas as mining bitcoin. It's in my opinion, one of the biggest email cloggers I have yet to experience. I have the rest of my life to see one that is worse, but challenge accepted, I guess.I've been on this app since I was under 18 and I'm 27 now. I have spent many an hour clearing these LinkedIn emails, and it's to the point where I fully understand that it could potentially be causing a detriment of heat to the earth due to the amount of emails they send out to literally anybody who is on planet earth and has an account. You will get carpal tunnel from swiping these emails into archive.
+There were no significant changes in AED levels or electroencephalographic findings
+A priori stratification by APOE genotype identified 27 families that had at least 1 member with AD whose genotype did not contain an APOE*4 allele
+In order to understand and model cochlear processes involved in TEOAE generation, it needs to be known whether the presence of normal hearing leads automatically to generation of TEOAEs
+A double-blind, crossover design was used to compare the effects of methylphenidate versus placebo on an electroencephalogram, AED levels, and the CPT
+Signals have qualitative and quantitative aspects
+I log in to see that my account was banned and have to verify??? That's never happened to me before and there's no reason stating what "suspicious activity" is being done? Even to get help as in a support ticket, or via email, you have to sign in but I CAN'T because my account is banned for no apparent reason! You guys need to seriously state the specific reasons as to why accounts are banned because I haven't done anything but use your guys clarify app to get verified and also apply to jobs and post my degree so what's the reason I'm so called "banned" for? Your customer service is horrible as well.. they're should definitely be an easier way to contact someone other than being a member because you ban us for no reason and it's impossible getting a hold of you guys. At that, it doesn't even state WHEN our accounts will be off banned mode or how long it takes for them to verify. It's ridiculous. I wouldn't recommend this app.
+BCR-ABL+ cells treated with ceramide also showed a rapid and sequential increase in the tyrosine phosphorylation of p210 (BCR-ABL), p46-56SHC and p120Cbl
+These initial observations are an encouraging step toward the description of non-HLA genetic susceptibility to MS
+Horrible interface
+I downloaded this app on my phone just so i could advise everyone not to use this website. For a senior project in my AP english class, everyone needs to create a LinkedIn account. Everything was going fine and I had spent about 3 hours working on the profile and only had about an hour or less of work left to do. The next day, I go to log in and there is an error message that says I need to submit ID due to suspicious activity. I talk to my teacher and this seems to be a problem with many other students. She contacts LinkedIn and they said it was due to a mass login on the same internet, and it will be fixed soon. The next day, very few students were still locked out, including me. I submitted my drivers license. The day after that, I was one of two students locked out. Skip a few days (3), and they FINALLY respond to me saying my account is permanently banned. I am now drastically behind in my class and this program is going to ultimately lead to my failure in this college class. I hope you guys fix your support team and maybe look at issues a little deeper before banning people.
+Home care workers face an increasing risk for workplace violence
+DESIGN : Cross sectional survey
+Histopathological findings included plaque-like encasement of the leaflets and chordal structures with intact valve architecture
+They have been combined with a nonlinear regression analysis method and tested on both simulated and real sets of biomedical MRS data selected with respect to the main problems usually encountered in quantifying biomedical MRS, specifically `` chemical noise, '' resulting from overlapping resonances, and baseline distortion
+It's driving me bananas but if you search for a job and put one of the filters on (company, date posted, industry, function etc) sometimes if you put a second filter on, it will remove the first filter. Please please please fix this. I have an iPhone 13 Pro Max and am running the newest IOS software. It is driving me crazy.Also!! When you go to select "company" there are many companies that you are able to choose from or you can search for one, you should make one of the defaults be the company the person is currently at or ones that they search a lot not random jobs. It doesn't make sense why the default ones aren't based on frequency. I don't understand how such a powerful and renown company can have so many bugs with their searching features. Hire more engineers. It's unfortunate there is nothing like LinkedIn because I would love to use anything that doesn't make me want to rip my hair out.
+Racist platform - sells your ID on the Dark Web
+The production of 1 greatly accelerates the decomposition of peroxynitrite to give nitrate
+Great app but...
+However, some recovery was seen for a change in the medium (growth medium to conditioned medium) up to 5-6 h postirradiation
+Love LinkedIn! But the app...
+Love LinkedIn - hate this one tiny bug
+These results indicate that the present parthenogenesis mechanism shares the same characteristics as that achieved by post-GVBD activation in the suppression of polar body formation as a key means for successful starfish parthenogenesis
+MAIN OUTCOME STUDY FACTORS : Cerebrovascular accident first-week fatality rate ; age- and sex-related incidence
+A dose-response was demonstrated for both opioids
+Horrible Interface
+Atrocious customer service
+I swear this site has provided me with so much stress just trying to set it up. I didn't dropped out of college and I left my job but they are both in my interests and I cannot change them. Whenever I open the app it tells me to finish updating my profile but I don't have anymore information to add. There's no way to say no. When I tried to leave things blank because they don't apply to me I couldn't. I got a notice that I had shown up in a search but when I opened the notification I was taken to a page within the app that did not work without a laptop. Why would this happen on the mobile app? It makes no sense. I've made all the connections I want to for now but every time I open the app I have to skip through the page to make connections with random people I don't know. This app and the website are infuriating to attempt to use. I use all kinds of other social media that work at least decently. This platform is by far the most angering to use. Maybe the team is overworked or has no time to fix these issues, but they are too glaring and long lasting to still be around.
+Successful arthrodesis was achieved in all five feet in patients with rheumatoid arthritis
+After adjusting for stage, factors such as poverty, other socioeconomic conditions, and treatment did not further explain the remaining survival deficit
+Few data exist as to whether there are clinical differences between prostate cancers that are inherited and those that occur in the general population
+The study was controlled for other factors that could influence the parasite 's prevalence, such as the availability of intermediate hosts
+DESIGN : A complete sequence analysis of the BRCA1 coding sequence and flanking intronic regions was performed in 798 women in a collaborative effort involving institutions from the United States, Italy, Germany, Finland, and Switzerland
+These results are consistent with two pools of DSBs (or cells), each with their own t (rep)
+Miracidia clearly preferred SCW from their specific host snail versus other sympatric snail species and did not respond to water conditioned with fish, tadpoles, or leeches
+However, the association of parvovirus and fetal pleural or pericardial effusions has not been reported
+Terrible tech support as a professional platform
+My LI app will no longer run, I'm forced to upgrade the app, but my iPhone doesn't support the new minimum de for the latest app. Sad LI made that decision to not support earlier sw versions :( and being in transition spending $1k on new phone isn't the smartest idea right now. Laptop only going forward :(Still abandoned- their support said delete app and reload, there was hope as when I was reloading it said that it would load the sw that was compatible with my phone and that some features may not work. Ha, who knew the feature that wouldn't work was the app itself... they have a screen that comes up stating "update for the latest sw" but you can't get out of that loop, my phone doesn't support latest iPhone sw so it can't get the latest LI download. A silly loop they could correct if they wanted, I guess they don't want to, they closed my case because I didn't respond back in the hour, like all I have to do is mess with this app, ugh. Now you know why I feel abandoned...
+Subjectively, patients improved from an average of 0.69 before arthrodesis to 4.89 after arthrodesis
+Out of all the Options
+RESULTS : Prevalence of confirmed infection was 2.6% (95% confidence interval 1.6% to 3.6%) in all women
+The newest update has them limiting your preferences for your job search an forcibly showing you state wide jobs like that helps. If someone like myself has set up job preferences why take them away and why start showing jobs to them that are outside of their preferences like a job search isn't already hard enough and tiring enough. Now I'm every time I open the app instead of being able to scroll through the recommendations I have to create different searches because the jobs recommended to me are 8+ hours away and don't interest me in the slightest. Just because I live in a state doesn't mean I'm comfortable commuting far and doesn't mean I'm trying to relocate to the opposite side of it. So who's dumb idea was it to add that useless and annoying feature that shows jobs to people outside their now very limited preferences. I don't want to be shown state wide jobs and according to the customer service rep it's a feature I can't manually get rid of. Dumbest update to date.
+I downloaded the app again because I thought it would be easier to use on my phone. I've been using it for about 30 minutes and it has crashed 3 times. It crashed while I was sending a request to connect. I was attaching a message to this request because the person doesn't know me. Now, It has sent the request and I can't finish my message. I don't know if it attached what I typed already or not. Given that LinkedIn is a tool to help people network and make professional connections, this is more than frustrating. Now, to reach out to this person, I would have to pay for a premium account and use "in mail." I'll not be using the app anymore which is a serious inconvenience. I stopped using the Linked In app years ago because it would lead to accidental connection requests while scrolling through searches or suggested connection.
+Here we show that individual amino-acid enantiomers from Murchison are enriched in 15N relative to their terrestrial counterparts, so confirming an extraterrestrial source for an L-enantiomer excess in the Solar System that may predate the origin of life on the Earth
+I used a linked in free profile for a while and when I was on the job market decided to try premium. I had it for a few months past the free trial, used it multiple times a week, but soon realized I didn't see any difference in the experience. No better prospects, never got any interest in my profile even though I opted to let recruiters see me. I even optimized my profile based on suggestions and no discernible difference in engagement or connections. Decided to cancel premium but then found out that I couldn't go back to my free profile (perhaps even they know there is no difference!) I had to erase my whole profile just to stop paying for a service that offered no real service! It is annoying because now I can't even use LinkedIn links (like when someone send me a job I might be interested in) without making a new free profile.
+Results of early rehabilitation seem to be influenced by duration and etiology of deafness, experience with hearing aids, and other factors
+The article stresses behavioral treatment modalities that eliminate the debilitating phobia
+When I first joined LinkedIn I had support help me consolidate two accounts and that wasn't an issue. Then because I debated some racists and debunked their lies, LinkedIn decided I was the guilty party and suspended my account. They demanded a copy of my ID front and back and/or passport. My identity could've been verified through my contacts who are high level attorneys, CEOs and even the Chamber of Commerce of which I am a member and LinkedIn sees that. They refused to reinstate my account. They're racists as revealed by an article in NY Times in 2019! I then created another account and rebuilt my profile. After a while LinkedIn seeing I was getting a lot of traction to my posts restricted my account again this time claiming I had two accounts AND I NEEDED to provide a passport or ID again! These people are insane! Suddenly my credit report warned that my email address was compromised and was found on the dark web! Guess who did this? Yes, LinkedIn. This company is a racist lying bunch of thieves and you should really avoid it at all cost. It's no longer the professional platform it once was.
+CONCLUSION : Calcium supplementation during pregnancy is associated with lower systolic blood pressure in the offspring, particularly among overweight children
+The absolute difference was 0.44 percent, with a one-sided 95 percent upper boundary of 1.49 percent, which exceeded the prespecified upper limit of 0.40 percent to indicate equivalence in 30-day mortality between the two regimens
+Needs improvement
+Primary efficacy endpoint was the percentage improvement in Unified Parkinson 's Disease Rating Scale (UPDRS) motor score
+< TO_SEE > CONTEXT : There is a substantial risk of a second cancer for persons with hereditary retinoblastoma, which is enhanced by radiotherapy
+One step forward, ten steps back
+INTERPRETATION : G-CSF treatment was associated with improved clinical outcome of foot infection in diabetic patients
+Social Connect NOT for CAREER seekers or EMPLOYMENT hopefuls
+Will not save profile picture
+INTERVENTION : After random assignment to 1 of 3 groups, the adolescents listened to a standardized audiotape depiction of an office visit during which they heard a physician who assured unconditional confidentiality, a physician who assured conditional confidentiality, or a physician who did not mention confidentiality
+Updating my review (Jan 15, 2022): Everything below PLUS app freezes when trying to post or upload a photo. (Yes, I deleted and reinstalled.) No such thing as tech help from what I can see. Just consumer forums and FAQs. Down to 1 star and deleting from my phone. --I used to enjoy this app, but it seems that it is getting less and less user-friendly on iPad and iPhone. For example, the landing page and profile page do not mirror the most important features that are on the laptop version. It would be nice to be able to easily view my recent updates and make changes to all of my profile sections--not just certain ones--from the app. A new thing I've noticed now is that some articles or links can't be opened in browser or forwarded through email/text. The "forward" arrow icon pops up with a "cancel" message instead and no other option. So weird. Sadly, the "app support" is just a forum of somewhat useless info, which is why I'm posting here. Hopefully the designers will read everyone's feedback on App Store and correct these issues in next update.
+A subchorial placental hematoma, which detected as a subchorial placental lucencies by ultrasonography, can be a cause of reversible nonimmune hydrops fetalis
+The author presents background information, including incidence and etiology
+FAMILIES : A total of 74 North American families with hereditary prostate cancer
+Cocaine-positive women presented at an earlier gestational age (32 weeks ', (24-34) vs. 33 weeks ', (23-36), p = 0.02) and had a significantly longer membrane rupture to delivery interval than women with a negative urine drug screen (174 hr, (6-475) vs. 33 hours (1-833), p = 0.01)
+In contrast, purified primary normal progenitors maintained under the same conditions die rapidly
+This platform is no longer useful for finding a job. The app does all of what it can to keep the user locked in to the feed section. If I'm reviewing a job and my screen lock becomes active (or view something off of the app), when I return to the app, I am immediately taken to my feed. When I go back to the job section, the job I was reviewing is no longer visible. I'm sure it's not deleted. It's just hard to tell which one I was viewing, especially if you're searching for similar jobs. It's like as soon as you put your phone down, you lose place in line. It creates nothing but wasted time.For anyone who has experienced a layoff, unemployment is a lot of work and it can easily wear you out. The last thing a person needs is a tool that slows them down. Of course a person can look elsewhere for jobs but why even have a job section if it has basic flaws.
+Neutrophils from the peripheral blood of these participants and from healthy controls were stimulated with opsonised zymosan, and superoxide production was measured by a spectrophotometric assay (reduction of ferricytochrome C)
+CONCLUSIONS : Using logistic regression analysis, we provide a method for evaluating the probability of a woman 's carrying a deleterious BRCA1 mutation for a wide range of cases, which can be an important tool for clinicians as they incorporate genetic susceptibility testing into their medical practice
+The effects of alanine and glycine substitution for tryptophan upon the species heterogeneity of gramicidin A analogs incorporated into SDS micelles have been investigated
+This article discusses dental anxiety and phobia
+College students spend 4 hr weekly on the PM & # 38 ; R consultation service as team members under resident supervision
+There was also a slightly higher rate of intracranial hemorrhage with the double-bolus method
+There were no differences among groups in sedation scores or side effects
+I've wasted my time for years with Indeed trying to search for jobs, and most of the wasted time is related to having to get through the pile of unrelated jobs thrown at me without any relevance to my job search. It doesn't matter how deeply I attempt to tweak my search, get rid of unrelated jobs or even supply comments to this company as to why their jobs are irrelevant - they keep on sending jobs that have absolutely no connection to my search criteria. Indeed could apply a significant fix by limiting job results to only those which have the exact match in the job title, but that's not an option. Why not? Because Indeed gets paid for the number of times a job shows up in a search, not whether it has no relevance. "Failed algorithms which waste job seekers time" - I suspect that this is the vision statement for Indeed. I depend on LinkedIn instead, and so should everyone else: Same jobs, but less wasted time and effort.
+We conducted a survey and audit of thermal equipment use in very low-birth-weight infants in five Ohio neonatal intensive care units (NICUs) to document regional practice
+INTERPRETATION : There is no treatment of choice for early-stage cervical carcinoma in terms of overall or disease-free survival
+Responding to recruiters new feature is terrible
+In our study, we specifically analyzed the role played by advanced maternal age and sonographically discovered abnormalities in the detection of autosomal trisomies
+343 eligible patients were randomised : 172 to surgery and 171 to radical radiotherapy
+For pain relief, ENS was significantly better (P & # 60 ; 0.05) than EMS ; but for the improvement of ROM, EMS was significantly better (P & # 60 ; 0.05) than ENS
+An account is basically required by potential employers if you're applying to white collar jobs, but is incredibly difficult to sort thru or actually get a response from companies- especially if you are applying thru linkedin itself. Would recommend only using the app to find companies that sound interesting, vetting them thru glassdoor, then applying via the company website itself.Suggestion 1 for better UX: limit applications to 100 per job with job listings closing automatically upon hitting 100 apps received. I am tired of seeing open job listings with 100+ applicants because i KNOW that the hiring managers are only skimming the top 5% of applicants from the pile and then underpaying them for their experience. Suggestion 2 for better UX: Years of required experience being listed as a filter within job listings search. I am not interested in a job that pays entry level wages or says "fast paced environment" and then requires 5-7 years of experience AND a bachelors.
+Chromosome 12 gave the strongest and most consistent results with a peak multipoint MLS of 3.5, suggesting that this region contains a new susceptibility gene for AD
+For spontaneous major haemorrhage (not procedure related), 1-year life-table estimates were 0.74% (0.43-1.1) in the 160 mg aspirin group and 1.4% (0.94-1.8) in the 3 mg warfarin with 80 mg aspirin group (p = 0.014 log rank on follow-up)
+linkedin (and the rest of social media) has become terrible
+Only sort of useful
+The present study set out to establish in a large sample if any ears could be found that lacked TEOAEs despite normal hearing threshold levels (HTL)
+These results provide electrophysiological evidence for early semantic activation and later phonological encoding
+Hallux length was well maintained after surgery with bone grafting, but it was more difficult to obtain the alignment goals
+Demographic information, hours from rupture of membranes to delivery, gestational age, and birth weight at delivery were compared using Fisher 's exact, Mann-Whitney U, and randomization tests where appropriate
+In the case of the 12 base-pair DNA duplex, torsion-angle molecular dynamics had a success rate of 52% while Cartesian molecular dynamics and metric-matrix distance geometry always failed
+Notification Issues
+Very buggy app
+What is going on !!!!
+These findings suggest growth factor deprivation and treatment with TNF-alpha or ceramide trigger different initial events both of which can lead to apoptosis in factor-dependent hematopoietic cells
+I was just trying to apply to a job on LinkedIn. They said to contact the employer by email and provided the email. When I press and hold LinkedIn does not allow you to select text to copy and paste into your mail. The email is also not a link I can click on to email. I unfortunately have to manually type in the long email letter by letter going back and fourth. Please fix this. I was also recently searching for jobs and reading a number of job descriptions. Many were in my field but did not apply to me. I would like to be able to hide them so I don't see them in the future. Don't see a way to do this. I only see Apply and Save buttons; no hide. When viewing a clients portfolio web link and clicking on it nothing happens within LinkedIn. This link should open up in Safari; I can not copy/paste it into Safari either and have to go back and fourth typing letter by letter.
+The urine screen did not influence management decisions
+Granulocyte-colony stimulating factor (G-CSF) increases the release of neutrophils from the bone marrow and improves neutrophil function
+The app for fake people
+The patient 's ability to wear shoes improved from 0.87 to 3.1
+106, 252-270, 1986) and double-strand break (DSB) (J
+Longer duration of illness (> 5 days) was associated with increased return within six weeks (hazard ratio 2.90, 1.70 to 4.92)
+Interaction effects between DR2 and chromosome 19q13.2 or MBP in MS susceptibility were not apparent
+I need more dots on my die!
+App crashes and crashes again.
+CONCLUSION : Methylphenidate is effective in treating children with epilepsy and ADHD and safe in children who are seizure free
+< TO_SEE > FINDINGS : G-CSF therapy was associated with earlier eradication of pathogens from the infected ulcer (median 4 (range 2-10) vs 8 (2-79) days in the placebo group ; p = 0.02), quicker resolution of cellulitis < (7 (5-20) vs 12 (5-93) days ; p = 0.03), shorter hospital stay (10 (7-31) vs 17.5 (9-100) days ; p = 0.02), and a shorter duration of intravenous antibiotic treatment (8.5 (5-30) vs 14.5 (8-63) days ; p = 0.02)
+This study was designed to determine and compare the dose-response characteristics, speed of onset, and relative potency of single-dose epidural fentanyl (F) and sufentanil (S) for postoperative pain relief
+LinkedIn... I love your platform. I have already posted a comment on the lack of a sort by "recent" function on for iOS. When I'm on my phone, the work around is to go to your mobile site, then request the desktop site, then select "sort by: recent". The most recent update you just pushed out now has "lives" or something that feels like a clone of Instagram/YouTube/Snapchat/Facebook, etc. I do not subscribe to these platforms because I believe LinkedIn is the only social media platform of any substance. There is quality content and networking happening everyday. Why did your newest update add a cookie-cutter "lives" function and not a simple "sort by:recent" function. Why am I seeing 3 week old posts with recent posts scattered about. Totally ridiculous... The one platform I thought was doing a service to a community of professional people is slowly becoming just another social media platform.
+Useful but opportunities for improvement
+High-dose fluoride treatment did not improve, but decreased, bone strength in rabbits, even in the absence of impaired mineralization
+RESULTS : Mean follow up time was similar (antibiotic group 1.07 years, other two groups 1.03 years)
+Dye labeled primers are used for cycle sequencing and an ABI PRISM 377 DNA sequencing system for analysis
+Not able to access on cellular, only WiFi
+Fluctuations in ambient temperature are inversely correlated to changes in AFI
+Messaging has become terrible and extremely frustrating
+After using the application for quite a while that I have added too many connections that I have never talked to. So recently I was trying to clean up those connections. Because I didn't find any option to delete all unused connections all at once so I just manually deleted them. All of sudden I received an system notification that I was using a third party software and restricted for log in in 1 hour. I was so frustrated that I never used anything like that and I was just simply cleaning up my unused connections. I mean if LinkedIn has the option to let me delete them all at once then I don't need to manually delete one by one and create a 'high volume' as they claimed. It is the inaccurate algorithms that mark me as something called 'using the the third party software'. In fact I don't even know there exists this kind of third party software. The customer representative was like this is a done deal that you are using the 3rd party software and any tickets regarding to this would be closed automatically. So LinkedIn made a mistake that they don't admit it. And the customer has no right to point it out! See this is the way how they treat their customers!
+LinkedIn closed my account for absolutely no reason.After I've updated the Linkedin App, I was asked to re-login, but instead of logging into my existing account, it created a new profile for me, completely blank. I was confused at first but was still able to search up my old account and thought of it as a tech glitch. I reached out to the service team and waited a few days before reaching out again, after they said they'll transfer my case to another team. By now I can no longer find my old account and am panicking, since it contains all my connections and the profile that I've built long and hard for.The team responded saying my account is restricted permanently for misrepresenting my identity, after I've provided my ID, and closed my case with no other details. Now I'm stuck with no information and a brand new account and no access to any of my past connections. I have offered the team to provide any relevant info to verify my past work/volunteering experiences, but I am just greatly disappointed by LinkedIn tech support.I've been patience since the start and comply with the rules, despite the great inconvenience that this has caused me work-wise, but I'm simply frustrated now that I may never get my account back for absolutely no reason. Apparently this has been an issue for many people and I don't understand why it continues to be one, being one of the most professional platform used by thousands.
+LinkedIn is my preferred way to interact with business professionals. I do, however, wish that the site would allow a longer time to deactivate an account or allow one to restore their account within a year. I deactivated my account for a "spiritual hiatus" and attempted to set my account back up and was unable to retrieve it. I built a great network before and now it seems like I am not getting as many recommendations for people that I knew from college and other businesses. I'm sure my post sounds like a combination of great things and not so great things, but I do really appreciate the fact that the tool is much more professional than most social media outlets.
+The effectiveness of treatment was assessed by conducting three measurements on each muscle before and immediately after treatment : subjective pain intensity ((PI) with a visual analog scale), pressure pain threshold ((PT) with algometry), and range of motion ((ROM) with a goniometer) of upper trapezius muscle (lateral bending of cervical spine to the opposite side)
+Not able to open an app on my iPhone
+LinkedIn was okay until I became a creator. It takes courage to post but over time, like 3-6 months, it becomes easier. At that point it becomes a lot more fun and now I look forward to using LinkedIn and this app every day. Not sure what to write? Use the 2 year, 2 day, or even 2 hour rule (credit Nicolas Cole). Just post what you would have benefited from had you known it 2 years ago, 2 days ago, or even 2 minutes ago!Developers: thanks for your work on the app!! First request: wish the scheduled posts could be edited. Second request: wish commenting was just better organized for conversations. Like replying to a reply to a reply. Also wish that the best answers/comments filtered to the top. Would draw you in as a user to know that the first comments are the most valued by the community (when there are dozens of comments and you don't want to sift through them all). Again thanks though for the great app!
+MAIN OUTCOME MEASURES : Mean blood pressure and rate of high blood pressure of children
+Chronological content feed - second attempt.
+F 100 and 200 microg and S 20 and 30 microg all achieved VAS scores & # 60 ; 10 mm in all women with no differences in time to 50% reduction in VAS (mean 11-16 min) and no differences in duration of analgesia (mean 117-138 min)
+I love LinkedIn, but would like to see an unlimited word count. There are so many times that I have a post that exceeds the allotted word count and it's frustrating. I could condense what i need to say most times, but sometimes I need to be able to say more than what LinkedIn allows. If the post is good, people will read it. If not, they don't have to read it and can just scroll on past. I'm sure if you took a poll from other LinkedIn users, you would find that most of them would agree that LinkedIn needs an unlimited word count. Don't limit it solely for the purpose of making sure people's posts are shorter. Give people the opportunity to add as much as they need to their posts. Sometimes you need to be able to add more to your post to properly say what you need to say because tightening your word count doesn't always cut it.
+ONLY use for a self propped social site to highlight career accomplishments, this is in NO way a site assisting with career, employment, job, or training for such in any way. I don't believe anyone in it's history actually found or obtained relevant employment by association with this site. The app is disgraceful with countless & continuous problems, the functions, features & customizations are only there to frustrate since no real functionality, majority if not all content from being contacted will be not or scam derived with no reportable solution for removal. If you have the time to waste, money to flush for the Pro version yet love advertisement inserts along with pro paying versions, lack of responsive support services & never getting much in return then you Will absolutely adore & promote this bug infested nightmare...there's way better available if a serious job seeker, career hopeful & educated enough to avoid this mess individual or business looking for hires. Hope this fades from App Store listings & internet history quickly for everyone's sake, the world could only improve for it doing so.
+The app is solid and very easy to use. My complaints are the content in the Feed and that the option "improve my feed" really offers no real options except adding new people to follow. It should be flexible and let you choose what you want to see more of OR less of. For example, in my feed I don't need to see every single time a connection "likes" something. I also remember I always used to see when my current connections connected to new people. I used to see those a lot every day. That was helpful info if a contact connected to someone new in my industry. It worked well in conjunction with the other "people I may know" info. I almost never see those types of updates anymore in my feed. Can you explain why the feed offers me zero flexibility to see the things that are most important to me?
+Prior attendance with upper respiratory conditions was also associated with increased reattendance
+There was no difference between groups in early return (13/238 (5.5%) v 27/437 (6%)), or complications (2/236 (0.8%) v 3/434 (0.7%))
+LinkedIn is the prime social networking app for professionals. Main avenue for jobs and all that.I think here's some opportunities for improvement:1. Make a filter for company size when looking for jobs. 2. A lot of recruiters post contracts as "full-time" as a sneaky way to get more applicants since they consider a full time role 40 hrs/week. Make this differentiation even clearer by saying "Permanent Full-Time." I don't know how many contract roles I've seen saying it's Full-Time and luckily I read every description pretty carefully.3. Make an option for when Looking For Work is turned on, we can put more details for recruiters to see. I get hit up for temp/contracts all the time and have no interest in them. Recruiters don't bother reading profiles.4. It would be great to have an option to rate or leave feedback on recruiters when you are communicating with them. Just like you have automatic "Yes/Not interested" responses, there should be a way to leave feedback on recruiter accounts since so many recruiters are rude, ghost candidates, post things unrelated to your field/interests, solicit, etc.
+Both right-sided and left-sided heart valves were involved
+They do not show you the jobs if you don't pay them $34 per month. Which is expensive, because the same jobs are also on google too. They have scammers who impose as HR people, even doing fake interviews. If you report this to LinkedIn, they do not recognize it as a scam. I had an interview where I needed to perform a difficult 2 days test and an interview where the boss was racketeering me and harassing me. I also found more complaints about this particular company on google and Reddit but LinkedIn says, there is nothing wrong with them. Out of 200 applications, maybe 30 will download your file but only 11 will open your file. It means their platform is full of fake jobs. I have been applying for jobs straight to the companies and I know for sure they were not seeking for anyone but LinkedIn would post a job from those companies. Scam, a lot of scam. They just want your money. I was better off without them
+Need Auto-Saving for Drafts
+While I have benefitted from certain services that LinkedIn provides such as LinkedIn learning, LinkedIn no longer appears to serve the individual user as much as it serves corporate recruiting efforts and corporate advertising.I work for a large educational system, which by this time has hundreds and hundreds of individual users with LinkedIn profiles, and while manually searching through a number of these profiles, the software app recently tried to force me into signing an agreement that I would voluntarily limit future searches through my own organizational network for others to connect with to a certain undisclosed, arbitrary number chosen by LinkedIn without any further information from LinkedIn to me or face suspension of my account by the app. The software has not allowed me any further access to my own profile until I sign that agreement. I strongly recommend avoiding this glorified recruitment service in favor of other newer, more personable services which allow the user to be more in control of his or her own searches.
+There was a significant correlation between the 2-, 3-, and 4-day mean temperature and AFI, with the 4-day mean being the most significant (r = 0.31, p & # 60 ; 0.001)
+All together, 27 fetuses had this disorder, representing 28.7% (27 of 94) of all cytogenetic aberrations detected at our center over the same period
+Hate speech from other LinkedIn members and employees
+Generally good, job search alerts broken
+Feel abandoned
+Needs improvement FIX FIX FIX
+Future studies should address the long-term consequences of prolonged use of fenfluramines
+The infant presented with extreme pallor, hypotonia, hepatosplenomegaly, and ascites
+In the absence of structural or karyotypic abnormalities, spontaneous resolution of the effusion portends for a successful pregnancy outcome
+LinkedIn excels as a premier platform for professional networking and career advancement, proving indispensable for job seekers and employers alike. It offers a rich array of features and a user-friendly interface that facilitates dynamic interactions and professional development. However, it could significantly benefit from enhanced privacy settings, allowing users to conceal the start dates of their career experiences to combat the prevalent issue of ageism in the professional realm. Just like in the education section of a user's profile, this added flexibility would enable individuals to showcase their skills and experiences without the constraint of timelines, promoting a more inclusive professional environment. Overall, LinkedIn stands as a powerful tool for career progression, but adopting more nuanced privacy features would further its commitment to fostering equitable and diverse professional communities.
+I'm a job seeker and I've been using LinkedIn to aid in my search. However, I'm afraid of missing a job opportunity as whenever I get a push notification I never hear any sound, unlike with my other apps. I always see a red number on my LinkedIn app widget showing me how many new notifications I have however I never hear any sound. All other sounds seem to be working which leads me to believe it's an app issue rather than a phone issue or operator error somewhere. Please help! Thanks!Also, unless I'm just missing it, I would LOVE the ability to select a custom notification sound within the app so I can identify a LinkedIn notification from all others. Thank you! Be safe!Oh! One thing I did forget to add is that I LOVE the section called "My Next Job Should Offer Me" where we're able to specify benefits we're asking for as well as the schedule we need. Oh and the Name Pronunciation feature is fantastic!!! Thank you!!!
+DESIGN : Randomized controlled trial
+I'm a huge fan of LinkedIn but Im unable to access anything right now!!! I am not unable to respond to my messages, or update anything....as soon as I try I am forced to open the app that at first said I didn't have an account so I reset my password..... even though I just had it open online.... now the app won't open because it's prompting me to fill out a page about my future job hopes and dreams that goes blank when I attempt to response. I don't understand why I can't use my online account anymore apps aren't dependable!! All I wanted to do is update my resume and read some articles! I apologize for my frustration I know it's an app thing! Please please please fix this issue!!!
+The treatment was for 30 days
+Changes to LinkedIn.
+OBJECTIVE : To estimate the crude incidence rates of cerebrovascular accidents among the black residents of Harare
+LinkedIn engages in profiling through barring certain searches
+When the effectiveness of treatment was compared with that of the placebo group (group A), there was significant improvement in PI and PT in group B (P & # 60 ; 0.01) but not in group C (P > 0.05)
+Please add an unlimited word count
+Atrocious Algorithms
+Our purpose was to systematically review data on brain serotonin neurotoxicity in animals treated with fenfluramines and the evidence linking fenfluramines to primary pulmonary hypertension (PPH)
+The app does its job in working the same as desktop. However, the notifications only work 1/10 times. In fact, most times you try to click an external link that will open the app, the app opens on an all black or all white screen and stays there. You have to force close the app to have it load normal again but if you were trying to see what the notification was about, I hope you have good memory. The app also does not hold your place in the feed so clicking anything for example a url will open their in-app window and returning will refresh your feed. Since LinkedIn is built on an algorithm to keep you interested in the app, you most likely won't ever see that post again as it generates new items it thinks would be of interest. Lastly the other night I had spent 20 minutes writing a long message regarding business with another user. I was on my last sentence when the app suddenly crashed. If you're wondering, no, the app did not draft my messages and I had to type everything back out again. Overall, it does its job but be prepared for a lot of bugs. And the notifications Never work.
+The addition of peroxynitrite to CO2 produces of the adduct ONOO-CO2- (1)
+I recently transferred Service from Verizon to AT&T. The LinkedIn app only works on Wi-Fi it does not work connected to cellular service. I have gone through all of the troubleshooting steps with AT&T as well as Apple. I'm on an iPhone 13 Pro. Everyone has said it is not an issue on their end that I should contact the developer. Please advise if there is some thing I can do to make this app push through on cellular service. The app is up-to-date with what's available in the App Store for me. I use this app for business regularly and I'm currently only able to use it on my home computer or I also cannot access the LinkedIn website on a browser unless I am The app is up-to-date with what's available in the App Store for me. I use this app for business regularly and I'm currently only able to use it on my home computer or my phone but must be connected to Wi-Fi. The strange part is on my phone I also cannot access the LinkedIn website on a browser unless I am connected to Wi-Fi. Please advise. Thank you.
+please read
+Unprofessional customer service and inaccurate algorithms
+Option to improve feed needs to be improved
+OBJECTIVES : Obesity is an important clinical problem, and the use of dexfenfluramine hydrochloride for weight reduction has been widely publicized since its approval by the Food and Drug Administration
+After being charged over $600 for a product I didn't realize I even bought due to their confusing interface I am being refused a refund and sent the same generic message about their no refund policy over and over again. I have used this website for years and am appalled by their dishonest practice. If you decide to ever use this website or app don't ever buy premium (it's useless) and don't ever put your card information on it or you might fall victim to what I'm going through. They make it too easy ti misinterpret something and buy something with an insane price tag without realizing what you're agreeing to. The app is also poorly designed as well. If you check Trust Pilot and the Better Business Bureau, they have a one star rating which is a huge red flag. Personally I would say avoid at all costs and don't ever give them any if your information except for the bare minimum. Don't expect a transparent app, business practice or to ever any kind of real conversation with customer service. Avoid at all cost but if you must use it Keep your expectations extremely low.
+I have used LinkedIn for the last time. There is so much effort in building a LinkedIn profile and like other texh giants," they" have the ability to suspend one's account. It may be automatic, it may be AI or it may be someone clicking buttons. It could have been me making a joke, asking, "What? No fact chexking?" to a post where someone said, "Wrong is wrong, even if everybody is doing it and right is right, even if nobody is doing it." Whatever the reason, they have shown me repeatedly that this is not a platform that I can trust or depend on when I need it. I resisted giving them my government issued ID to regain access to my account, the last time this happened and eventually, "I Loved Big Brother," like at the end of 1984. I sent themmy ID and they still did not unlock my account. The promose was broken and was just a lie. They feel that they have power over us but it only works if we cooperate.
+In a subset of 71 Ashkenazi Jewish women, only 2 distinct deleterious mutations were found : 185delAG in 17 cases and 5382insC in 7 cases
+multilocularis from wild hosts has been demonstrated, the question of its long-term efficacy and other unresolved problems have to be addressed by consecutive studies before routine application can be recommended
+Show me your papers! Repeatedly suspended.
+Ropinirole monotherapy is an effective and well-tolerated therapeutic option for treatment of early Parkinson 's disease
+Appetite suppressants-most commonly fenfluramines -increase the risk of developing PPH (odds ratio, 6.3), particularly when used for more than 3 months (odds ratio, > 20)
+Here we show that the medial temporal region is active during encoding and that the hemisphere activated and the amount of activation depend on the type of stimulus presented (objects or words), whether the stimulus can be encoded for meaning (real objects and words versus nonsense objects and words), and task experience (first versus the second time a task is performed)
+DESIGN : Retrospective case study in which clinical data were extracted from medical and pathological records
+Screening all women aged & # 60 ; or = 25 and all women who had had two or more partners in the past year would have detected 87% (20/23) of infections
+NOT for dedicated job seekers
+However, except for genes in the HLA system, specific susceptibility loci are unknown or unconfirmed
+Variation within the MBP locus on chromosome 18q23 showed no effect in MS
+The study was terminated at 30 min if satisfactory analgesia was not achieved
+Right temporal and cerebellar hemorrhages were present
+Zalak Patel is a consummate PROFESSIONAL
+BACKGROUND : Inadequate dietary intake of calcium and vitamin D may contribute to the high prevalence of osteoporosis among older persons
+Freezes, Can't apply for jobs, links don't work! NEEDS THE TO BE FIXED!
+Number one, if your application in the Apple IDs door would support a dialog box when selecting apps support I would be writing this in that dialog box and not under reviews. My review is based on the fact that Face ID is not used by LinkedIn. LinkedIn's responses leave your app signed in. I do not wish to do that, the information in the app is too private for me to take LinkedIn suggestion. The technology for Face ID has been incorporated to I would say 70 percent of the applications at this point, so it tells me that updating your app to use this functionality cannot be that difficult. I'll be waiting for the Face ID recognition in your app soon. I would use your out more by not having to manually put in my password, holding the phone to your face makes my motivation to use your application what greater than having to use my large fingers To type in my password. Thank you, John
+Reanalysis of the original data on DSB repair shows that they are consistent with two first-order repair rates (18 +/- 7 min and about 52 min)
+Please please please for all that is good and holy, please make it so when you connect with someone on the app, it pops up a "personalize invite" modal like it does on the desktop version. Or make the desktop version work like the mobile version (but the desktop flow is better IMO). The amount of times discrepancy between the two platforms has caused me to accidentally send an invite to early & blow a connection because I connect, want to personalize, and then can't, and then need to withdraw and wait 3 (THREE!!) weeks before I can resend is just bonkers. Please please please fix this and I will come back and make this review 5 stars and love you forever. LinkedIn is a great platform, and could be a lot bigger than it is with people spending more time on it. They don't because little stuff like this drives us wild.
+However, 63% of the trisomic fetuses belonged to mothers aged 35 years and above
+This clinical commentary discusses the current management of the syndrome including the Abbe-McIndoe procedure, the most commonly used method of surgical correction, and the Frank vaginal dilation method, the most common nonsurgical method of correction
+Sibpair analysis (n=54) resulted in maximum lod scores (MLSs) of 1.5, 2.6, 3.2, and 2.3 for these markers, with a peak multipoint MLS of 3.5
+Fifty-one per cent of stroke victims were below the age of 54 years
+The main thing I use LinkedIn for is messaging and the past couple of months with these updates the messaging has gone so downhill with more glitches and bugs than ever it makes me not even want to use it. Sometimes I click on a person to message and it just never pulls up the conversation or freezes when I am trying to hit send. Also, when you have a search going, it now disappears the moment you click out of the app to read a text or something which makes you have to keep putting in what you are searching for and it never used to do that. This app used to be great before they ruined it with pointless updates that only made things worse with no noticeable improvements. I would have rated 5 stars before these updates if that tells you how frustrating this has become.
+This study tested the hypothesis that to reduce the rate of macrosomic infants in gestational diabetes cases, good glycemic control should be initiated before 34 completed gestational weeks
+Thirty-five per cent of patients died within 1 week of the stroke
+This is a perfect example of what NOT to do in mobile app design. 1. Useless messenger button that does nothing but try and get you to download the app by popping not one, but two screens (if you don't have it). I won't load Facebook Messenger because it won't let me select no tracking without constantly being harassed. 2. Extremely small "x" for closing annoying ad windows so that if you have any normal or larger than normal finger (god forbid you try and use your thumb) then you end up opening the item instead. 3. Allowing you to block some advertisers for only 30-Days at a time??? Seriously. If it wasn't bad enough that you show me adds from advertisers I've already blocked, now you're unblocking them after 30 Days??4. Extremely small "x" for closing annoying recommended groups windows so that if you have any normal or larger than normal finger (god forbid you try and use your thumb) then you end up opening the item instead. Can I just get you to stop recommending stuff to me? I'm not sure what algorithm you use but the recommendations are horrible. 5. Translations are rarely ever correct. Give up here and just license Google's translator. I'm not saying Google Translate is good, but it would at least be correct more than your translations are. There are more, I'm sure. But we will stop here.
+In the present study, parthenogenotes were obtained by the activation of immature oocytes with caffeine before treatment with 1-methyladenine (1-MeAde) to induce oocyte maturation
+CONCLUSIONS : A complete genomic screen in families affected with late-onset AD identified 4 regions of interest after follow-up
+More of those initially prescribed antibiotics initially returned to the surgery with sore throat (38% v 27%, adjusted hazard ratio for return 1.39%, 95% confidence interval 1.03 to 1.89)
+These were matched for maternal age, parity, and race to twin gestations (N = 53) from the same population
+I am a blind person using the screen reader voiceover for iOS and I am having some serious VoiceOver accessibility problems. For one thing, in the new version, when I double tap on my Contacts, the app crashes every time. Secondly, when I am browsing through my people you may know section, the app get slower and slower and slower the longer eyebrows. Thirdly, if you were using voiceover, there is no option to remove someone from your people you may know list so that you do not see them anymore. The accessibility problems with this app used to be somewhat minimal. I am really looking forward to seeing this app fixed and using LinkedIn again. It is one of my favorite social networking platforms. I apologize if other people are bored by this review, but LinkedIn also does not have any where in the app or on the app support page to provide feedback about the app. I would also be kind of nice if developer of LinkedIn provided details about its updates in its release notes, but many App developers don't do this. It's just a pet peeve.
+Similar B/W differences were observed for patients with advanced disease but not with early stage
+For additional evaluation, females were divided into two subgroups, osteoporotic or nonosteoporotic, based on x-rays and lumbar bone mineral density (BMD) as measured by qCT
+METHODS : The effect of roxithromycin was assessed in a double-blind, randomised, prospective, multicentre, parallel-group, placebo-controlled pilot study of 202 patients with unstable angina or non-Q-wave myocardial infarction
+The advantages and pitfalls of this new method are discussed in the light of the results obtained with an infantile GSDII patient
+It is generally assumed that abiotic synthesis of amino acids on the early Earth resulted in racemic mixtures (L- and D-enantiomers in equal abundance)
+Horrible app/ Customer service
+Evidence for the presence of a serotonin1A (5-HT1A) receptor subtype in the salmonid fish brain has recently been presented
+At 293 sites, we randomly assigned 8803 patients who had had myocardial infarction, treatment with 160 mg aspirin, 3 mg warfarin with 80 mg aspirin, or 1 mg warfarin with 80 mg aspirin
+On admission, patients were randomly assigned G-CSF (filgrastim) therapy (n = 20) or placebo (n = 20) for 7 days
+SETTING : Cullen Eye Institute, Baylor College of Medicine, Houston, Texas, USA
+Women in the cocaine positive group were of higher parity (3, (0-7) vs. 1, (0-6), p = 0.001) and tended to be older (27, (23-42) vs. 25, (14-40))
+Transesophageal echocardiography demonstrated aliasing on color flow mapping in the left main coronary artery in 1 case and proximal right coronary artery in the other, along with severely depressed left ventricular anterior wall and right ventricular function, respectively
+Fluoride increased the vertebral BV/TV by 35% (P & # 60 ; 0.05) and tibial ash weight by 10% (P & # 60 ; 0.05)
+iPhone X
+PARTICIPANTS : Institutions selected 798 persons representing families (1 person for each family) thought to be at elevated a priori risk of BRCA1 mutation due to potential risk factors, such as multiple cases of breast cancer, early age of breast cancer diagnosis, and cases of ovarian cancer
+After 7 days ' treatment, neutrophil superoxide production was significantly higher in the G-CSF group than in the placebo group (16.1 (4.2-24.2) vs 7.3 (2.1-11.5) nmol per 10 (6) neutrophils in 30 min ; p & # 60 ; 0.0001)
+Curtis, Radiat
+The objective of this study was to determine if the rate of preeclampsia is increased in triplet as compared to twin gestations
+This is one super cool, fast paced, never boring but it does make you want shout! Absolutely one of the best games I've played online ever. This game wants you to advance extremely fast, quick and easy. But it also shows you how to manipulate, destroy and leave one of your friends broken, broke and mad because I chose to leave their building's destroyed so they have to fix them. That costs more money, money is a big deal in this game. Everything costs money and breaking into your opponents bank and stealing their life's savings is a bit evil, but I had fun doing it and it was worth the risk. The farther I put him back the further I can get a head.
+This app is really frustrating to use. When looking at a company's open roles, I can't filter by location because it will just reset the search to all jobs in that location, not restricting to the company whose page I was just on. If I try to restrict further by searching the company name, all these unrelated jobs show up in the search. This doesn't happen on desktop so I'm not sure why it's happening on mobile. Also the filters aren't very functional - so many jobs that are included in the "entry level" filter but require 7+ years experience, or are marked as fully remote but are actually fully in person. The jobs I'm recommended aren't even close to the field or function of my current roles. My home feed is just full of fake inspirational posts about CEOs deigning to give their employees breadcrumbs and people foaming at the mouth over their "generosity". It's not a good tool anymore.
+SETTING : Clinic populations in the continental United States
+When each group was divided into two additional subgroups based on the initial PI, it was found that ENS could reduce PI and increase PT significantly (P & # 60 ; 0.05), but did not significantly (P > 0.05) improve ROM, as compared with the placebo group for both subgroups
+Sequelae include severe developmental delay and asymmetric double hemiplegia
+< TO_SEE > METHODS : We identified valvular heart diseasein 24 women treated with fenfluramine-phentermine who had no history of cardiac disease
+Sera were drawn 12 days from ovulation in non-IVF cycles or 9 days after ET and were assayed for PIBF and beta human chorionic gondotropin
+Update: LinkedIn heard my prayers and fixed my app. I no longer get kicked out when trying to send direct messages, and the app works generally faster. Thank you so much for giving me the opportunity to utilize this awesome app to its fullest potential again! Let me start out by saying that LinkedIn has changed my life in so many positive ways. I found my new career through the app, and started down a journey of positive personal development when I rejoined the platform. Hence the four stars. I needed to take one star away because, at least in the app, it almost always crashes and boots me out when I try to send a Direct Message to someone. I end up needing to go in and retype the message 2-3 times before it will send without crashing and closing out. I know plenty of other people are experiencing the same problem, because they post about it. Please fix this issue, I love this platform for so many reasons, but this is frustrating! Thank you in advance for your attention and all that you do to make LinkedIn the wonderful place that it is! Update: my app has worked perfectly since the last update. No more getting booted out when sending DMs. Thank you LinkedIn!
+Until a few days ago I was able to open and converse through my LinkedIn app on my iPhone 7 Plus now all of a sudden it doesn't open up I've deleted it I've installed updates on my iPhone I've done everything it said and every time I download the LinkedIn app it won't open up I don't know what's going on I can access it on a regular computer but for some reason I can't access it through the app on my iPhone I don't know if I'm having the same issue as the person before me that wrote about not being connected to Wi-Fi Not being able to access itbut I was able to even get itWithout Wi-Fi about a week ago I don't understand what's going on
+The most common indication leading to diagnosis was abnormal ultrasound finding (48.2%), followed by advanced maternal age (44.4%)
+The manufacturer claims that moxonidine `` accurately targets imidazoline receptors at the cardiovascular control centre in the brainstem '' and is `` as effective as current first-line therapies for essential hypertension ``
+Need more options
+A radiation risk for all sarcomas combined was evident at doses above 5 Gy, rising to 10.7-fold at doses of 60 Gy or greater (P & # 60 ;.05)
+However, careful visual assessment of the recorded waveforms for the remaining ears did not unequivocally show absence of TEOAE characteristics in any ear with normal HTLs
+The average postoperative metatarsophalangeal dorsiflexion angle was 15.6 degrees and the first metatarsophalangeal angle was 3.1 degrees
+CONCLUSIONS : Double-bolus alteplase was not shown to be equivalent, according to the prespecified criteria, to accelerated infusion with regard to 30-day mortality
+OBJECTIVE : To investigate several loci spanning 3 candidate regions for a role in multiple sclerosis (MS) susceptibility in 2 ethnic groups using both single-locus and haplotype analyses
+The involved upper trapezius muscles were treated with three different methods according to a random assignment : group A muscles (n = 18) were given placebo treatment (control group) ; group B muscles (n = 20) were treated with electrical nerve stimulation (ENS) therapy ; and group C muscles (n = 22) were given electrical muscle stimulation (EMS) therapy
+An LRP was only observed on no-go trials when the semantic (not the phonological) decision determined the response hand
+I love this app however it frustrates me that the current job title seems to be the only factor that is included when recruiters are delivered search results. As a consultant, my title can be anything and it may be quite different from my functional role and area of expertise. Depending upon the currently listed job title, I've noticed a sharp difference in the number contacts I receive from recruiters and the roles they are seeking. I wish that other factors such as education, certifications and skills would be considered when recruiters are delivered search results. Also, LinkedIn should allow subscribers to search by certification or certificate. Currently a search for common certifications returns no results despite the fact that these certifications are listed within job descriptions.
+Eight women also had newly documented pulmonary hypertension
+The daily high temperature ranged from 71 to 104 degrees F and AFI values ranged from 1.7 to 24.7 cm during the study period
+I spent about an hour carefully crafting a reply to a message, had assembled a bunch of links and useful information for the recipient, and had to navigate away from the app momentarily to grab a link. When I switched back to LinkedIn, the app opened to the main landing page and when I reopened the message I had been composing a reply to, the text field was empty. I lost an hour of work and I feel so frustrated, distraught, and disheartened. Auto-save for in-progress messages is pretty basic functionality for most messaging programs and if your app had had it, I wouldn't be in this situation. I wish I could setup a LinkedIn auto-response that directs everyone who messages through LinkedIn me to contact me on a different interface, so that I could avoid using LinkedIn's messaging tool entirely. Objectively, the app is basically fine aside from this issue and does not really deserve only one star. I'm just still reeling from the emotional blow of this experience and looking for an outlet for my frustration as well as to share this feedback with you while it's fresh and in a way that will help your team feel the pain that can be caused by this product gap and really understand the impact. Thanks for reading.
+Synergism in tumorigenesis and independent segregation of the MSI phenotype suggest that Msh2 and p53 are not genetically epistatic
+The rate of macrosomic and large-for-gestational-age infants were 5 and 11%, respectively, in the early group as compared to 25 and 29% in the `` late '' group (p & # 60 ; 0.05)
+Warning about LinkedIn Terrible
+Professional Networking Tool
+The rehabilitation consultation service is particularly effective as an introduction to hospital-based medical practice due to the diagnostic variety of the patients, the functional approach of rehabilitation, and student exposure to multiple hospital settings
+The rate of severe preeclampsia was increased significantly in the triplet group 12 of 53 (22.6%) as compared with the twin group 3 of 53 (5.7%) (OR = 4.9, 95% CI 1.2-23.5, p = 0.02)
+Not a safe place to have a professional opinion in most professions. Especially if you are a women. LinkedIn does not respect women whom have professional scientific reasoning and professional opinions that vary from the political slant posted by either a member or otherwise linkedin; however LinkedIn will continue to push the political slant into your feed continually inviting you to acknowledge the political topic which is slanted and not academically or professionally sound. If you don't like it and if you contribute your intelligent, professional and academically sound comment LinkedIn members state hate things to a person such as Members stating hate statement to me and others I have witnessed ). Very disappointed in the lack of professionalism from LinkedIn employees whom allow the members and themselves to be so hateful to women and others on LinkedIn whom state professional scientific academically sound verbiage on a topic.
+Account was Banned?
+Our data suggest that the relative analgesic potency of epidural S : F is approximately 5 and that there are no differences between the opioids in the onset, duration, and effectiveness of analgesia when equianalgesic doses are administered postoperatively after lidocaine anesthesia for C/S
+When a job expires I can't see it on the app only on the actual website. There's to be a way to archive old jobs that I've applied to. When applying to jobs externally it reroutes you to a different web browser entirely so you don't lose your application. Is there a way to add quick swipe left so when there's a job that you're not interested in you can hide it?It would also be nice to add a filter to where you can check "see only see United States jobs available for anyone in the United States and the state you live in for remote jobs instead of the app showing for example "Chicago, IL United States" and you can apply for these jobs because the job poster only wants candidates from that area state. This would make job searching a lot faster.
+INTERPRETATION : Antichlamydial antibiotics may be useful in therapeutic intervention in addition to standard medication in patients with coronary-artery disease
+1-year life-table estimates for the primary event were 8.6% (95% CI 7.6-9.6) for 160 mg aspirin, 8.4% (7.4-9.4) for 3 mg warfarin with 80 mg aspirin, and 8.8% (7.6-10) for 1 mg warfarin with 80 mg aspirin
+Paid experience is same as free experience
+Twenty-four patients with cerebral palsy and 24 control subjects between the ages of 4 and 12 yr were enrolled in this study
+This study examined the aggressiveness of colon tumors in blacks and whites to explore its role in the racial survival differences
+In the `` early '' and `` late '' groups, mean gestational age at the beginning of treatment was 30.0 +/- 3.8 and 36.2 +/- 1.2 weeks, and duration of treatment was 9.6 +/- 4.1 and 3.7 +/- 1.8 weeks, respectively
+In fact, time course studies demonstrated an even faster onset of apoptosis in ceramide-treated BCR-ABL+ BAF3 cells as compared to normal controls
+after being a paying premium linkedin member for years, my account was suddenly deleted by linkedin simply because i chose to talk openly with others on it about being treated for sickness i've dealt with that was related to the moderna vaccine - and trying to spread helpful information to others who may experience the same issue. i'm also a single parent trying to find work ASAP after the past two years of struggle. linkedin has been deleting actual doctors who have been trying to help with this stuff as well. it's clear the people in linkedin's trust & safety department don't care about any of this and have joined the insane groupthink cult governing too many of these companies now, censoring anyone who might displease their advertisers, even to their userbase's detriment. the fact that linkedin has started using engagement and suppression algorithms like twitter and facebook is making it an increasingly more toxic environment as well. i've switched to indeed. linkedin has become absolutely inhumane and will sink its own ship if this is how it continues to conduct business.
+But the origin of life required, owing to conformational constraints, the almost exclusive selection of either L- or D-enantiomers, and the question of why living systems on the Earth consist of L-enantiomers rather than D-enantiomers is unresolved
+All hand instruments, handpieces and triplex syringes were autoclaved between patients
+The present study also investigates the possible participation of cytoplasmic receptors in the signal transduction pathway, using cotransfection experiments and a new approach that selectively determines the contribution of cytoplasmic receptors in the process of signal transduction
+CONTEXT : A mutation in the BRCA1 gene may confer substantial risk for breast and/or ovarian cancer
+CONCLUSIONS : Families that provide evidence for segregation of an altered HPC1 gene are characterized by multiple cases of prostate cancer that, in most respects, are indistinguishable from nonhereditary cases
+In the past when I was contacted by a recruiter in my message inbox, I could respond directly to the recruiter. Now, the app forces you to respond whether or not you are interested in the job before you have a chance to respond. To me this is a 100% unnecessary feature and is an unprofessional way to contact a recruiter to express interest. The right way to treat a recruiter is to write that person directly and let them know your intentions, perhaps including some background for your decision. This can help keep communication channels open in the future and is a networking opportunity. This new feature is a result of someone "overthinking" the situation. This goes back to the old saying, "If its not broke, don't fix it"
+The human DNA mismatch repair (MMR) genes are highly conserved homologues of the Escherichia coli MutHLS system, which contribute to genomic stability by surveillance and repair of replication misincorporation errors and exogenous DNA damage
+Overall, the age-specific rates for both sexes rose with age, with the rates for women being higher at all age strata except for the group 45-54 years
+This partial recovery is also parallel to the partial activation of JAK2, indicating that the signal transduction pathway implicated JAK2
+The objective of this article is to explore attitudes of an inner-city, pregnant cohort about general and HIV-related prenatal care
+Horrible
+CONTEXT : Approximately 9% of prostate cancer cases have been estimated to result from inheritance of mutated prostate cancer susceptibility genes
+I had a LinkedIn account that I ran my photography business through messaging & sharing with customers sharing links they would send me and me to them. After at least five years of using the App LinkedIn stopped letting me log into my account asking for verification on KYC which I provided after about two months and the account still not able to login. I inquiring as to why the account was shutdown and not letting me login. LinkedIn finally has sent me an email saying that I have shared post or links they don't want being forwarded. No warnings in five years of being loyal to the brand and they kicked me to the curve like I an ugly dog to them. It really hurt my business and feelings that this is as good as it gets in business today use at your own risk don't put nothing on the platform you want to keep because I was locked out. I shared a few Crypto links?
+The absolute worst networking app...
+There was no significant difference of mean latency, amplitude, or amplitude ratio of SSR between the two groups under electric stimulus, startling stimulus, and deep breathing conditions
+Ropinirole-treated patients had a significantly greater percentage improvement in UPDRS motor score than patients who received placebo (+24% vs -3% ; p & # 60 ; 0.001)
+RESULTS : In 102 women (12.8%), clearly deleterious mutations were detected
+It is not known if such damage occurs in humans or if there are clinical consequences
+App is still terrible, customer service is worse
+Furthermore, when all deliveries were analyzed, regardless of risk status but limited to gestational age > or = 36 weeks, the rates did not change (12.6%, 280 of 2214 ; primary 9.2%, 183 of 1994)
+Seven basic considerations can be identified that determine the evidence contained in a signal : quantitative strength of the association, consistency of the data, exposure response relationship, biological plausibility, experimental findings, possible analogies and the nature and quality of the data
+As the ambient temperature increases, there is an increase in insensible fluid loss and the potential for dehydration
+The heart valves had a glistening white appearance
+Side effects were recorded
+The mean training time per week was 3.2 +/- 0.8 h. In the entire group of female dancers, no significant effects of dancing on radial or lumbar bone density could be observed
+In the age of TikTok and Instgram, still bans and kicks you off for no reason, but spam accounts selling test prep are fine. Apparently if you are looking for jobs and networking and you're using the app they will "ban" or "restrict" your account with no explanation except for "we've detected automated activity" I'm using the linkedin app on my iPhone! Why the developers don't implement some features to deter actual automated activity such as: -Have users take a selfie, like Uber Driver app does -Remind users they have been scrolling for too long, like tiktok does -Tell users what the "threshold" for use is instead of having a "blackbox" and then issuing a ban. This is probably the worse customer experience/ user experience in an app ever! Should we use your app or not? Do we use your app and then get banned for no reason!? Linkedin Dev team please address this!!
+METHODS : 40 diabetic patients with foot infections were enrolled in a double-blind placebo-controlled study
+Antibiotics prescribed for sore throat during the previous year had an additional effect (hazard ratio 1.69, 1.20 to 2.37)
+The Biopsychosocial Model of medical practice is demonstrated through multiple interdisciplinary perspectives of needs and interventions for patients with obvious functional deficits
+Great app. Unsubscribe from emails before Earth Dies.
+Nutritional assessment, substrate immunonutrition, and disease specific issues are presented
+Better as a creator
+The respective rates of any stroke and of hemorrhagic stroke were 1.92 and 1.12 percent after double-bolus alteplase, as compared with 1.53 and 0.81 percent after an accelerated infusion of alteplase (P=0.24 and P=0.23, respectively)
+I have tried on my web browser on my laptop and in the LinkedIn App, and the system will not allow me to save a profile picture. It will let me upload a photo, edit it with cropping and filters, and hit the "save" button, but the photo does not save. In the web browser, the "save" grays out after I press it and does nothing else. If I then close the edit window, it warns that my changes have not been saved and gives only the option to discard changes and close the window or to cancel and return to the editing screen. In the App, if I press save, it comes up with a "saving" message and the wheel spins for about 3 seconds, then the entire app crashes and shuts down. Pretty worthless app if it doesn't even function well enough to put in a profile picture. My other limited experiences with the app have also been negative. When trying to edit education history and work history, the app is clunky and sometimes slow to respond. It is also hard to predict where in your profile the info you are entering is going to actually show.
+They are the most thorough platform available, full of many different types of new beginnings. However, I have not had quality customer support experiences with them. I have been trying to utilize a benefit that they offer veterans for some time now, and something is not working, but when I reach out to them for assistance, I get very little other than a vague response. I have contacted them on several occasions, but it's still a mystery to me why this opportunity continues to elude me. I have given up hope, and I rarely use (or try to) the site anymore because of this. It makes me sad because I don't have the resources and abilities I once possessed, and this is the best place for me to focus on what I CAN do and learn about new opportunities. I was looking forward to this being the beginning of the most lucrative and rewarding career in my lifetime. Yet, I have hit another stumbling block instead.
+Extremely buggy on iOS on iPhone 12 max. I've deleted the app and reinstalled. There's some very badly managed API calls that feel like it's using jQuery instead of a modern MVC framework causing every other scroll to click instead. I don't understand why it became so much worse, or how QA engineers let this out the door. It's very badly managed developers and software. This was not an issue at all a couple years ago, shortly after Microsoft bought LinkedIn. It is however unsurprising that the UI became much worse after MS got involved and MS has no design cycle nor do they have a QA cycle so it's absolutely likely the UI will continue to get worse. Sigh, hopefully I'll land a job so I don't need to use this broken and frustrating UI much longer. The most frustrating part it I tried to tell LinkedIn and the customer support claimed the couldn't replicate, which is funny because I tested on several iPhones. It's gaslighting on their part. The whole customer support experience also reminded e of Microsoft, where thy just don't believe you and assume the customer is doing something wrong. I don't think they want to fix it.
+CONCLUSIONS : The significant chromosome 19q13.2 single-locus and multilocus haplotype associations with MS in Caucasian and Chinese patient samples indicate an effect from a nearby disease susceptibility locus
+LinkedIn is the absolute worst, corrupt, and non user friendly social app out there. There are too many reasons to even begin mentioning all of them. Just to name one: if a user wants to unfriend or leave a group, LinkedIn intentionally makes it impossible to do so on the mobile app.I'm also still dealing with a "glitch" that the app had a few years ago. Without giving any permission to access my iPhone's contacts, LinkedIn automatically dumped every one of my LinkedIn contacts into my iPhone's contacts. Well over 500 contacts were combined with my personal contacts. Not a big deal you say? Well those contacts birthdays were then automatically also added to iPhone's calendar. Everyday I would have at least a couple of these birthdays showing. For well over a year I was manually removing these contacts one name at a time. LinkedIn admitted to the error. Thanks that did nothing to help. So at this point I have zero trust in the integrity, honesty, or reliability of LinkedIn.
+Most adverse experiences were related to peripheral dopaminergic activity
+I like the app. I recently started trying to use the job alert feature. I searched for a job ("software engineer") at a mid-sized clothing retailer. I have seen open roles there before, and I know they employ software developers, but they don't happen to have any open roles right now. I would like to create an alert for when such a role does open up, but the "turn on alerts for this search" toggle only appears for searches that currently produce results. Please add the toggle to all job searches, no matter how specific, since this is the point of these alerts-to let users know if/when a relevant job pops up. Also there's no obvious "feedback" area in the app, so I had to come here. :(
+The initial hemoglobin was 2.2 g/dL, the Kleihauer-Betke stain was 27.6% (highest level ever reported)
+Prevalence on the basis of enzyme immunoassay was 1.6% (0.8% to 2.7%), with a sensitivity of 60% and a specificity of 100%
+I hired Zalak to design my site based on two criteria: 1 She had a very fine portfolio. 2. More importantly, she asked me for my business, then asked again, then asked again. I said yes and for the next year, was never sorry for a single moment. First, she is a highly skilled and wonderful designer with an eye for art that is hard to find. 2. She has common sense and approaches her design as a customer centered business understanding that in the end, I needed to be satisfied and happy with her product, for her to be happy with her job; 3. I can absolutely trust her with my site, including passwords, and project funding (she was not only a designer but a great project manager). 4. She set me up with her own programmers in which I paid her and she paid them. She guaranteed them and she got what she wanted from the without damaging the site. (Something I had a hard time doing). I found Zalak to be pleasant and personable. She was always on time and always dependable and always did what she told me she would do. I have hired dozens of talented off-shore programmers/designers and few do I give a recommendation and none do I give a full throated recommendation to, except to Zalak. Call me any time in Missouri to talk to me.
+It could reduce PI significantly more (P & # 60 ; 0.05) than placebo controls only for the subgroup with mild to moderate pain, but not with severe pain
+RESULTS : PIBF was detected in 29.5% of non-pregnant women and 52.5% of pregnant women
+API calls make UI stall
+Transiently evolked otoacoustic emissions (TEOAE) have been reported in several studies as absent in a small minority of normal ears
+I too am very disappointed with LinkedIn and the direction it has taken. This previously was a great site for people to communicate with other people and companies. Whether it was trying to get a job, improving their position, finding a new workplace due to moving to a new area or trying to move up to a better job, talking with others about how to improve their job skills so they can prepare for a better job or keeping in touch with friends they worked with in the past or, even better, finding old friends and learning how to stay in touch with them. It WAS a great site. Now, as the previous commenter said, LinkedIn is now just another Facebook or Twitter. It's totally lost all the professionalism it started with. If anyone in power within the LinkedIn organization reads this comment, please, please, PLEASE change the format back to what the original people or person had as their vision. If you don't, I can foresee this site going down the tubes. I, for one, will do my best to start a new site with only one goal... to make a website for companies and individuals to get together and help each other grow both as a company and as individuals who want to help a company grow and a company willing to help their dedicated individuals grow within the company. Also, have learning institutions small and large contribute to the individuals growth.Is anyone out there with me on this?
+I've heard so many people speak highly of LinkedIn, so I tried it out. Unfortunately this app does NOT work, as it needs serious updates. When you try to apply for any of the job listing your are given the option to give permission to attach your LinkedIn profile in replace or your resume, it does NOT work. Once you click you get a white screen or when it does go through to the next screen where it will ask for your name and contact info it freezes. Therefore you can not apply for any of their job listings. So I look up the company on google to find their website and then apply that way with my own resume. The only thing LinkedIn is good for right now is to locate a job listing. Then apply on their site. I gave 1 star because at least they still proved job listing I can apply to on the companies web site directly.
+This article discusses many of the nutritional topics important to the intensivist
+The effect was most pronounced in the central part of the treated area, where no positive fox was found in the 2 months before the end of the trial
+It offers potential as a non-invasive screening tool
+The function of the short form was examined after cotransfection of both the long and short forms
+I honestly have never written a review before, but I left LinkedIn because of this when I was fond of building connections on LinkedIn. However, someone had to make me very uncomfortable. LinkedIn is for professional use. When posting, there should be a sensitivity filter so what ever comment or post is being shared on LinkedIn directly is not hurtful or disrespectful. I had a horrible experience with someone disrespecting my profession and the person said "in ms your considered a joke" on my post that involved my profession having an appreciation week. That is the total opposite of what LinkedIn in is used for.Update: it's sad to see that my review was overlooked and not cared about. Bullying or any form of making someone uncomfortable especially on a professional job app should be taken seriously. You should be able to post about your profession without someone trying to undermine a position. What a shame.Adding on to my review: I hope LinkedIn is able to talk with him, and give him a warning. It's sad because I did not want to leave LinkedIn, but because of this behavior that should not be tolerated I left and it was probably for the best. I just wanted to be recognized for all the hard work I do for my job and post it on my page and someone had to ruin it all. I know it's not you LinkedIn, but I hope you can help me.
+Trisomy 18 fetuses showed hydramnios, intrauterine growth retardation, microcephaly, spina bifida, and nonimmune hydrops fetalis
+Serious VoiceOver accessibility problems
+PARTICIPANTS : Two hundred and seventy-three 'first-ever ' strokes prospectively identified over a 12-month period
+Most of the content I've encountered on this app is fake. Fake people with their fake jobs and fake integrity awards. No. These are the lowest most untrustworthy people on the planet. Sorry but I can't play your charades. The job section is a hot mess. More fake advertisements than Indeed and that's hard to beat. Then you have a bunch of fake made of companies with their websites that took them under and hour to make. Are people still being fooled? Most of the jobs that are real are expired and outdated or closed. It just a stalker app. Provides no real business connections or opportunities.if you try to delete the profile they still want to advertise you for up to 2 more weeks. Sorry but when I delete you no longer have the option to do anything with my personal information. This app is useless. Just people spying on me or sending odd spam email. I can't wait until I figure out how to reclaim my information from these jerks!

docparse/extract.py

@@ -0,0 +1,721 @@
+"""One entry point for every format: extract(path) -> document dict.
+
+Document shape (JSON-serialisable):
+    {"source", "format", "metadata", "pages": int, "blocks": [...], "sheets": [...], "warnings": [...]}
+Blocks, in reading order, are one of
+    heading   {text, level}
+    paragraph {text}
+    list_item {text, level?}
+    table     {rows: [[cell, ...], ...]}
+    figure    {}                       (pictures; their text is read only with ocr_images=True)
+    header / footer {text}             (Word page headers and footers)
+Every block can carry page, bbox [x0, y0, x1, y1] (PDF points or image pixels), sheet, source
+("text-layer", "ocr", "structure") and, for OCR output, confidence (mean per-character probability).
+"""
+import csv
+import datetime as dt
+import html
+import io
+import re
+import shutil
+import subprocess
+import tempfile
+from functools import lru_cache
+from pathlib import Path
+
+import numpy as np
+
+ROOT = Path(__file__).resolve().parents[1]
+DEFAULT_MODEL = ROOT / "artifacts" / "docparse" / "ocr.pt"
+
+IMAGE_EXT = {".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".webp", ".gif"}
+CONVERT = {".doc": "docx", ".odt": "docx", ".rtf": "docx", ".xls": "xlsx", ".ods": "xlsx",
+           ".ppt": "pdf", ".pptx": "pdf", ".odp": "pdf"}
+
+
+@lru_cache(maxsize=2)
+def recognizer(path=None):
+    from .ocr.model import Recognizer
+    path = Path(path or DEFAULT_MODEL)
+    if not path.exists():
+        raise FileNotFoundError(f"OCR weights not found at {path}. Train them with `docparse train-ocr`.")
+    return Recognizer(str(path))
+
+
+def extract(path, ocr="auto", ocr_images=False, model=None, dpi=200, password=None):
+    """ocr: "auto" (only pages without a text layer), "always", or "never"."""
+    path = Path(path)
+    ext = path.suffix.lower()
+    doc = {"source": path.name, "format": ext.lstrip("."), "metadata": {}, "pages": 0, "blocks": [],
+           "sheets": [], "warnings": []}
+    opts = {"ocr": ocr, "ocr_images": ocr_images, "model": model, "dpi": dpi, "password": password}
+    if ext in CONVERT:
+        with tempfile.TemporaryDirectory() as tmp:
+            converted = _libreoffice(path, CONVERT[ext], tmp)
+            doc["warnings"].append(f"converted {ext} to .{CONVERT[ext]} with LibreOffice before parsing")
+            _dispatch(converted, converted.suffix.lower(), doc, opts)
+    else:
+        _dispatch(path, ext, doc, opts)
+    return doc
+
+
+def _dispatch(path, ext, doc, opts):
+    if ext == ".pdf":
+        pdf(path, doc, **opts)
+    elif ext in IMAGE_EXT:
+        image(path, doc, opts["model"])
+    elif ext in (".docx", ".docm", ".dotx"):
+        docx(path, doc, opts["ocr_images"], opts["model"])
+    elif ext in (".xlsx", ".xlsm", ".xltx"):
+        xlsx(path, doc, opts["ocr_images"], opts["model"])
+    elif ext in (".csv", ".tsv"):
+        delimited(path, doc)
+    elif ext in (".txt", ".md"):
+        text_file(path, doc)
+    else:
+        raise ValueError(f"unsupported file type: {ext}")
+
+
+def _libreoffice(path, target, out_dir):
+    exe = shutil.which("soffice") or shutil.which("libreoffice")
+    if not exe:
+        raise RuntimeError(f"{path.suffix} files need LibreOffice (soffice) for conversion")
+    subprocess.run([exe, "--headless", "--convert-to", target, "--outdir", out_dir, str(path)],
+                   check=True, capture_output=True, timeout=180)
+    out = Path(out_dir) / (path.stem + "." + target)
+    if not out.exists():
+        raise RuntimeError(f"LibreOffice could not convert {path.name}")
+    return out
+
+
+# ---------- images ----------
+
+def image(path, doc, model=None):
+    from PIL import Image, ImageSequence
+    from .ocr.page import analyze
+    rec = recognizer(model)
+    img = Image.open(path)
+    frames = [f.copy() for f in ImageSequence.Iterator(img)] if getattr(img, "n_frames", 1) > 1 else [img]
+    doc["metadata"].update({"width": img.width, "height": img.height, "mode": img.mode})
+    for n, frame in enumerate(frames, 1):
+        doc.setdefault("page_sizes", []).append([frame.width, frame.height])
+        blocks, meta = analyze(frame, rec)
+        mark_page_furniture(blocks, frame.height)
+        for b in blocks:
+            b.update(page=n, source="ocr")
+        doc["blocks"] += blocks
+        doc["metadata"].setdefault("ocr", []).append({"page": n, **meta})
+    doc["pages"] = len(frames)
+
+
+# ---------- PDF ----------
+
+def pdf(path, doc, ocr="auto", ocr_images=False, model=None, dpi=200, password=None):
+    import pdfplumber
+    with pdfplumber.open(path, password=password) as pdf_:
+        doc["metadata"].update({k: str(v) for k, v in (pdf_.metadata or {}).items() if isinstance(v, (str, int, float))})
+        doc["pages"] = len(pdf_.pages)
+        for n, page in enumerate(pdf_.pages, 1):
+            doc.setdefault("page_sizes", []).append([float(page.width), float(page.height)])
+            chars = page.chars
+            scanned = len("".join(c["text"] for c in chars).strip()) < 20
+            if ocr == "always" or (ocr == "auto" and scanned):
+                if ocr == "never":
+                    continue
+                blocks = _ocr_pdf_page(path, n - 1, page, model, dpi, password)
+            else:
+                blocks = _pdf_text_page(page)
+                if ocr_images:
+                    _ocr_pdf_figures(path, n - 1, blocks, model, dpi, password)
+                if scanned:
+                    doc["warnings"].append(f"page {n} has no text layer; run with ocr='auto' to read it")
+            mark_page_furniture(blocks, float(page.height))
+            for b in blocks:
+                b["page"] = n
+            doc["blocks"] += blocks
+            try:
+                links = [h.get("uri") for h in page.hyperlinks if h.get("uri")]
+            except (UnicodeDecodeError, ValueError, KeyError, TypeError):  # malformed annotations: skip links
+                links = []
+                doc["warnings"].append(f"page {n}: hyperlink annotations could not be decoded")
+            if links:
+                doc["metadata"].setdefault("links", []).extend({"page": n, "uri": u} for u in links)
+
+
+def _pdf_text_page(page):
+    from .ocr.page import group, text_blocks
+    S = 2.0  # mask pixels per PDF point
+    W, H = int(page.width * S) + 2, int(page.height * S) + 2
+    tables, candidates = [], []
+    for t in page.find_tables():
+        candidates.append(t.bbox)
+        rows = [[_clean_cell(c) for c in row] for row in _split_stacked_rows(t.extract())]
+        cells = [c for row in rows for c in row]
+        # Boxes drawn around code or text frames also look like grids; real tables are mostly filled.
+        if len(rows) >= 2 and max(len(r) for r in rows) >= 2 and sum(1 for c in cells if c) >= 0.5 * len(cells):
+            tables.append((t.bbox, rows))
+    tboxes = [b for b, _ in tables]
+
+    def in_table(w):
+        cx, cy = (w["x0"] + w["x1"]) / 2, (w["top"] + w["bottom"]) / 2
+        return any(x0 - 1 <= cx <= x1 + 1 and y0 - 1 <= cy <= y1 + 1 for x0, y0, x1, y1 in tboxes)
+    all_words = [w for w in page.extract_words(extra_attrs=["size", "fontname", "upright"], use_text_flow=False,
+                                               keep_blank_chars=False) if not in_table(w) and w["text"].strip()]
+    # Rotated text (margin stamps, vertical axis labels) would line up with every body line and look
+    # like a table column; it is kept out of the layout pass and reported as its own block.
+    figures = _vector_figures(page, candidates, all_words)
+    figures = [f for f in figures if not any(g != f and _inside(f, g) for g in figures)]
+    if any(_inside(b, f) for b in tboxes for f in figures):  # a "table" drawn inside a chart is part of it
+        tables = [(b, r) for b, r in tables if not any(_inside(b, f) for f in figures)]
+        tboxes = [b for b, _ in tables]
+        all_words = [w for w in page.extract_words(extra_attrs=["size", "fontname", "upright"], use_text_flow=False,
+                                                   keep_blank_chars=False) if not in_table(w) and w["text"].strip()]
+
+    def in_figure(w):
+        cx, cy = (w["x0"] + w["x1"]) / 2, (w["top"] + w["bottom"]) / 2
+        return next((k for k, (x0, y0, x1, y1) in enumerate(figures) if x0 <= cx <= x1 and y0 <= cy <= y1), None)
+    fig_words = [[] for _ in figures]
+    for w in all_words:
+        k = in_figure(w)
+        if k is not None:
+            fig_words[k].append(w)
+    all_words = [w for w in all_words if in_figure(w) is None]
+    words = [w for w in all_words if w["upright"]]
+    rotated = [w for w in all_words if not w["upright"]]
+    mask = np.zeros((H, W), bool)
+    boxes = []
+    for w in words:
+        h = w["bottom"] - w["top"]
+        y0, y1 = int((w["top"] + 0.18 * h) * S), int((w["bottom"] - 0.18 * h) * S) + 1
+        x0, x1 = int(w["x0"] * S), int(w["x1"] * S) + 1
+        y0, y1 = max(0, min(y0, H - 1)), max(1, min(y1, H))
+        x0, x1 = max(0, min(x0, W - 1)), max(1, min(x1, W))
+        mask[y0:y1, x0:x1] = True
+        boxes.append((y0, y1, x0, x1, w))
+    sizes = [w["size"] for w in words]
+    body = float(np.median(sizes)) if sizes else 10.0
+
+    def read(segs):
+        out = []
+        for y0, y1, x0, x1 in segs:
+            ws = [b[4] for b in boxes if y0 <= (b[0] + b[1]) / 2 <= y1 and x0 <= (b[2] + b[3]) / 2 <= x1]
+            out.append((_styled_text(ws, body), None))
+        return out
+    th = body * 0.64 * S  # height of a shrunken word box at body size
+    blocks, th = text_blocks(mask, th, read, figures=False)
+    for (x0, y0, x1, y1), rows in tables:
+        blocks.append({"kind": "table", "box": (int(y0 * S), int(y1 * S), int(x0 * S), int(x1 * S)), "rows": rows,
+                       "confidence": None})
+    for im in page.images:
+        x0, y0, x1, y1 = im["x0"], im["top"], im["x1"], im["bottom"]
+        if (x1 - x0 >= 10 and y1 - y0 >= 10 and (x1 - x0) * (y1 - y0) > 400
+                and not any(_inside((x0, y0, x1, y1), f) for f in figures)):
+            blocks.append({"kind": "figure", "box": (int(y0 * S), int(y1 * S), int(x0 * S), int(x1 * S))})
+    for (x0, y0, x1, y1), ws in zip(figures, fig_words):
+        blocks.append({"kind": "figure", "box": (int(y0 * S), int(y1 * S), int(x0 * S), int(x1 * S)),
+                       "text": " ".join(w["text"] for w in sorted(ws, key=lambda w: (round(w["top"]), w["x0"])))})
+    out = group(blocks, th)
+    if rotated:
+        x0, y0 = min(w["x0"] for w in rotated), min(w["top"] for w in rotated)
+        x1, y1 = max(w["x1"] for w in rotated), max(w["bottom"] for w in rotated)
+        text = " ".join(w["text"] for w in sorted(rotated, key=lambda w: (w["x0"], -w["bottom"])))
+        out.append({"type": "paragraph", "text": text, "rotated": True,
+                    "box": (int(y0 * S), int(y1 * S), int(x0 * S), int(x1 * S))})
+    for b in out:
+        y0, y1, x0, x1 = b.pop("box")
+        b["bbox"] = [round(x0 / S, 1), round(y0 / S, 1), round(x1 / S, 1), round(y1 / S, 1)]
+        b["source"] = "text-layer"
+        b.pop("confidence", None)
+        _pdf_heading_by_font(b, words, body)
+    return out
+
+
+def _inside(a, b, tol=2.0):
+    return a[0] >= b[0] - tol and a[1] >= b[1] - tol and a[2] <= b[2] + tol and a[3] <= b[3] + tol
+
+
+def _overlap(a, b):
+    w = min(a[2], b[2]) - max(a[0], b[0])
+    h = min(a[3], b[3]) - max(a[1], b[1])
+    return w * h if w > 0 and h > 0 else 0.0
+
+
+def _vector_figures(page, tables, words):
+    """Charts and diagrams drawn with vector graphics (not embedded images).
+
+    Drawing primitives are rasterised onto a coarse grid, dilated and grouped into connected
+    regions. A region is a figure when it is large, holds enough primitives and little text
+    (shaded tables and boxed text are dense with words; charts and diagrams are not)."""
+    from .ocr.page import components
+    W, H = float(page.width), float(page.height)
+    S = 0.5  # grid cells per point
+    prims = []
+    for o in page.curves + page.lines + page.rects:
+        x0, y0, x1, y1 = o["x0"], o["top"], o["x1"], o["bottom"]
+        if (x1 - x0) * (y1 - y0) > 0.5 * W * H or any(_inside((x0, y0, x1, y1), t, 6) for t in tables):
+            continue  # page backgrounds and table rules
+        prims.append((x0, y0, x1, y1, o["object_type"] == "curve"))
+    if len(prims) < 4:
+        return []
+    grid = np.zeros((int(H * S) + 2, int(W * S) + 2), bool)
+    pad = 6
+    for x0, y0, x1, y1, _ in prims:
+        grid[max(0, int((y0 - pad) * S)):int((y1 + pad) * S) + 1, max(0, int((x0 - pad) * S)):int((x1 + pad) * S) + 1] = True
+    rows, starts, ends, groups = components(grid)
+    out = []
+    for g in groups:
+        x0, x1 = starts[g].min() / S + pad, ends[g].max() / S - pad
+        y0, y1 = rows[g].min() / S + pad, (rows[g].max() + 1) / S - pad
+        if x1 - x0 < 50 or y1 - y0 < 40 or (x1 - x0) * (y1 - y0) < 0.01 * W * H:
+            continue
+        inside = [p for p in prims if _inside(p[:4], (x0, y0, x1, y1), 4)]
+        curves = sum(p[4] for p in inside)
+        if len(inside) < 6 and curves < 2:
+            continue
+        text_area = sum((w["x1"] - w["x0"]) * (w["bottom"] - w["top"]) for w in words
+                        if _inside((w["x0"], w["top"], w["x1"], w["bottom"]), (x0, y0, x1, y1)))
+        if text_area > 0.2 * (x1 - x0) * (y1 - y0):
+            continue
+        area = (x1 - x0) * (y1 - y0)
+        if any(_overlap((x0, y0, x1, y1), t) > 0.3 * area for t in tables):
+            continue  # the rules of a table (pdfplumber's box can be a little smaller than the grid)
+        out.append((x0, y0, x1, y1))
+    return out
+
+
+def _font_style(w):
+    name = w["fontname"].lower().split("+")[-1]
+    bold = any(k in name for k in ("bold", "black", "heavy", "semibold", ",b")) or name.endswith("-b")
+    italic = any(k in name for k in ("italic", "oblique", ",i")) or name.endswith("-i")
+    return bold, italic
+
+
+def _styled_text(ws, body):
+    """Words of one text segment -> text with Markdown emphasis and <sup>/<sub> from the PDF's fonts.
+
+    Characters in a different size come back as separate words ("x" "2"); they are joined without
+    a space when they touch, and marked as superscript or subscript from their baseline."""
+    if not ws:
+        return ""
+    lines = {}
+    for w in sorted(ws, key=lambda w: w["top"]):
+        key = next((k for k in lines if abs(k - w["bottom"]) < 0.6 * max(w["size"], body)
+                    or (w["top"] >= k - 0.9 * body and w["bottom"] <= k + 0.4 * body)), None)
+        lines.setdefault(w["bottom"] if key is None else key, []).append(w)
+    pieces = []  # (text, bold, italic, joined_to_previous)
+    for key in sorted(lines):
+        line = sorted(lines[key], key=lambda w: w["x0"])
+        big = max(w["size"] for w in line)
+        base = float(np.median([w["bottom"] for w in line if w["size"] >= 0.9 * big]))
+        prev = None
+        for w in line:
+            text = w["text"]
+            small = w["size"] < 0.85 * big
+            if small and w["bottom"] < base - 0.2 * w["size"]:
+                text = f"<sup>{text}</sup>"
+            elif small and w["bottom"] > base + 0.1 * w["size"]:
+                text = f"<sub>{text}</sub>"
+            touching = prev is not None and w["x0"] - prev["x1"] < 0.12 * big
+            pieces.append((text, *_font_style(w), touching))
+            prev = w
+    if all(p[1] for p in pieces):  # a fully bold segment is a heading or label: no inline marks
+        pieces = [(t, False, it, j) for t, _, it, j in pieces]
+    if all(p[2] for p in pieces):
+        pieces = [(t, b, False, j) for t, b, _, j in pieces]
+    out, i = [], 0
+    while i < len(pieces):
+        bold, italic = pieces[i][1], pieces[i][2]
+        j = i
+        run = pieces[i][0]
+        while j + 1 < len(pieces) and pieces[j + 1][1:3] == (bold, italic):
+            j += 1
+            run += ("" if pieces[j][3] else " ") + pieces[j][0]
+        mark = "**" if bold else ""
+        mark += "*" if italic else ""
+        out.append(("" if pieces[i][3] and out else " ") + f"{mark}{run}{mark[::-1]}")
+        i = j + 1
+    return "".join(out).strip()
+
+
+_INLINE_MARKS = re.compile(r"\*\*\*(.+?)\*\*\*|\*\*(.+?)\*\*|(?<![\w*])\*(?!\s)(.+?)(?<!\s)\*(?![\w*])|</?su[pb]>")
+
+
+def plain(text):
+    """Text without the inline Markdown marks added from PDF fonts."""
+    return _INLINE_MARKS.sub(lambda m: next((g for g in m.groups() if g is not None), ""), text)
+
+
+def _pdf_heading_by_font(b, words, body):
+    """Use real font sizes (not box heights) to decide headings in born-digital PDFs."""
+    if b["type"] not in ("paragraph", "heading"):
+        return
+    x0, y0, x1, y1 = b["bbox"]
+    ws = [w for w in words if x0 - 1 <= w["x0"] and w["x1"] <= x1 + 1 and y0 - 2 <= w["top"] and w["bottom"] <= y1 + 2]
+    if not ws:
+        return
+    size = float(np.median([w["size"] for w in ws]))
+    all_bold = all("bold" in w["fontname"].lower() for w in ws)
+    short = len(b["text"]) < 120
+    if len(b["text"]) < 200 and size >= body * 1.15:  # long titles are still titles when set large
+        b["type"], b["level"] = "heading", 1 if size >= body * 1.7 else 2 if size >= body * 1.35 else 3
+        b["text"] = plain(b["text"])
+    elif short and all_bold and len(ws) <= 14:
+        b["type"], b["level"] = "heading", 3
+        b["text"] = plain(b["text"])
+    elif b["type"] == "heading":
+        b["type"] = "paragraph"
+        b.pop("level", None)
+
+
+PAGE_NUMBER = re.compile(r"^(page|p\.?|pg\.?)?\s*[-–—(]?\s*([0-9]{1,4}|[ivxlcdm]{1,6})\s*[-–—)]?(\s*(of|/)\s*[0-9]{1,4})?$", re.I)
+
+
+def mark_page_furniture(blocks, page_height, band=0.08):
+    """Running headers and footers: short text blocks inside the top or bottom band of the page.
+    Headings stay in the body (a document title is often near the top of page one)."""
+    for b in blocks:
+        if b["type"] not in ("paragraph", "list_item") or not b.get("bbox") or len(b.get("text", "")) > 200:
+            continue
+        x0, y0, x1, y1 = b["bbox"]
+        if y1 <= band * page_height:
+            b["type"] = "header"
+        elif y0 >= (1 - band) * page_height:
+            b["type"] = "footer"
+        else:
+            continue
+        if PAGE_NUMBER.match(plain(b["text"]).strip()):
+            b["page_number"] = True
+
+
+def _split_stacked_rows(rows):
+    """Tables without horizontal rules come back from pdfplumber with several records stacked in one
+    row, one line each per cell ("8833\n8837\n8838"). When every filled cell of a row holds the same
+    number of lines, those lines are separate records: split them into rows."""
+    out = []
+    for row in rows:
+        parts = [str(c).split("\n") if c else [] for c in row]
+        counts = {len(p) for p in parts if p}
+        if len(counts) == 1 and (k := counts.pop()) > 1 and sum(1 for p in parts if p) >= 2:
+            out += [[p[i] if p else "" for p in parts] for i in range(k)]
+        else:
+            out.append(row)
+    return out
+
+
+def _clean_cell(c):
+    return "" if c is None else " ".join(str(c).split())
+
+
+def _render_pdf_page(path, index, dpi, password):
+    import pypdfium2 as pdfium
+    pdf_ = pdfium.PdfDocument(str(path), password=password)
+    try:
+        return pdf_[index].render(scale=dpi / 72).to_pil()
+    finally:
+        pdf_.close()
+
+
+def _ocr_pdf_page(path, index, page, model, dpi, password):
+    from .ocr.page import analyze
+    img = _render_pdf_page(path, index, dpi, password)
+    blocks, _ = analyze(img, recognizer(model))
+    k = 72 / dpi  # pixels -> PDF points
+    for b in blocks:
+        b["bbox"] = [round(v * k, 1) for v in b["bbox"]]
+        b["source"] = "ocr"
+    return blocks
+
+
+def _ocr_pdf_figures(path, index, blocks, model, dpi, password):
+    """Read text inside pictures on a born-digital page; it is stored on the figure block."""
+    from .ocr.page import analyze
+    figs = [b for b in blocks if b["type"] == "figure"]
+    if not figs:
+        return
+    img = _render_pdf_page(path, index, dpi, password)
+    k = dpi / 72
+    for f in figs:
+        x0, y0, x1, y1 = f["bbox"]
+        sub, _ = analyze(img.crop((int(x0 * k), int(y0 * k), int(x1 * k), int(y1 * k))), recognizer(model))
+        text = "\n".join(_block_text(b) for b in sub if _block_text(b))
+        if text:
+            f["text"] = text
+            f["text_source"] = "ocr"
+
+
+# ---------- Word ----------
+
+def docx(path, doc, ocr_images=False, model=None):
+    import docx as docx_
+    from docx.table import Table
+    from docx.text.paragraph import Paragraph
+    d = docx_.Document(str(path))
+    cp = d.core_properties
+    for k in ("title", "author", "subject", "keywords", "created", "modified", "last_modified_by"):
+        v = getattr(cp, k, None)
+        if v:
+            doc["metadata"][k] = v.isoformat() if isinstance(v, dt.datetime) else str(v)
+    seen = set()
+    for s in d.sections:
+        for kind, part in (("header", s.header), ("footer", s.footer)):
+            if part.is_linked_to_previous:
+                continue
+            text = "\n".join(p.text for p in part.paragraphs if p.text.strip())
+            if text and (kind, text) not in seen:
+                seen.add((kind, text))
+                doc["blocks"].append({"type": kind, "text": text, "source": "structure"})
+    for el in d.element.body.iterchildren():
+        tag = el.tag.rsplit("}", 1)[-1]
+        if tag == "p":
+            b = _docx_paragraph(Paragraph(el, d))
+            if b:
+                doc["blocks"].append(b)
+            for box in el.iter("{http://schemas.openxmlformats.org/wordprocessingml/2006/main}txbxContent"):
+                text = "\n".join(Paragraph(q, d).text for q in box.iterchildren() if q.tag.endswith("}p")).strip()
+                if text:  # text boxes and shapes are not part of paragraph.text
+                    doc["blocks"].append({"type": "paragraph", "text": text, "source": "structure", "text_box": True})
+        elif tag == "tbl":
+            doc["blocks"].append({"type": "table", "rows": _docx_table(Table(el, d)), "source": "structure"})
+    images = [r.target_part for r in d.part.rels.values() if "image" in r.reltype]
+    doc["metadata"]["images"] = len(images)
+    if ocr_images:
+        _ocr_blobs([im.blob for im in images], doc, model)
+    doc["pages"] = 1
+
+
+def _docx_paragraph(p):
+    text = p.text.strip()
+    if not text:
+        return None
+    style = (p.style.name if p.style is not None else "") or ""
+    b = {"type": "paragraph", "text": text, "source": "structure"}
+    low = style.lower()
+    if low == "title":
+        b.update(type="heading", level=1)
+    elif low.startswith("heading"):
+        digits = "".join(ch for ch in style if ch.isdigit())
+        b.update(type="heading", level=int(digits) if digits else 1)
+    else:
+        ppr = p._p.pPr
+        num = ppr.numPr if ppr is not None else None
+        if num is not None or "list" in low:
+            b["type"] = "list_item"
+            if num is not None and num.ilvl is not None:
+                b["level"] = int(num.ilvl.val)
+            elif style[-1:].isdigit():  # "List Bullet 2" -> second level
+                b["level"] = int(style[-1]) - 1
+            if "number" in low:
+                b["ordered"] = True
+    if b["type"] == "paragraph" and p.runs and all(r.bold for r in p.runs if r.text.strip()) and len(text) < 100:
+        b.update(type="heading", level=3)
+    return b
+
+
+def _docx_table(table):
+    rows = []
+    for row in table.rows:
+        out, last = [], None
+        for cell in row.cells:
+            if cell._tc is last:  # horizontally merged cells repeat the same element
+                out.append("")
+                continue
+            last = cell._tc
+            parts = [p.text for p in cell.paragraphs if p.text.strip()]
+            for inner in cell.tables:
+                parts += [" | ".join(r) for r in _docx_table(inner)]
+            out.append(" ".join(" ".join(parts).split()))
+        rows.append(out)
+    return rows
+
+
+def _ocr_blobs(blobs, doc, model):
+    from PIL import Image
+    from .ocr.page import analyze
+    for k, blob in enumerate(blobs, 1):
+        try:
+            img = Image.open(io.BytesIO(blob))
+            img.load()
+        except Exception:
+            doc["warnings"].append(f"embedded image {k} could not be decoded")
+            continue
+        blocks, _ = analyze(img, recognizer(model))
+        for b in blocks:
+            b.update(source="ocr", image=k)
+        doc["blocks"] += blocks
+
+
+# ---------- Excel / CSV ----------
+
+def _cell_value(v):
+    if isinstance(v, dt.datetime) and v.time() == dt.time(0):
+        return v.date().isoformat()
+    if isinstance(v, (dt.datetime, dt.date, dt.time)):
+        return v.isoformat()
+    if isinstance(v, float) and v.is_integer():
+        return int(v)
+    return v
+
+
+def _regions(grid):
+    """Split a sheet into blocks of cells separated by fully empty rows, then empty columns."""
+    def empty(v):
+        return v is None or (isinstance(v, str) and not v.strip())
+    if not grid:
+        return []
+    out = []
+    rows = [any(not empty(v) for v in r) for r in grid]
+    for r0, r1 in _true_runs(rows):
+        band = grid[r0:r1]
+        cols = [any(not empty(r[c]) for r in band) for c in range(len(band[0]))]
+        for c0, c1 in _true_runs(cols):
+            block = [r[c0:c1] for r in band]
+            keep = [any(not empty(v) for v in r) for r in block]
+            first, last = keep.index(True), len(keep) - keep[::-1].index(True)
+            out.append((r0 + first, c0, block[first:last]))
+    return out
+
+
+def _true_runs(flags):
+    runs, start = [], None
+    for i, f in enumerate(list(flags) + [False]):
+        if f and start is None:
+            start = i
+        elif not f and start is not None:
+            runs.append((start, i))
+            start = None
+    return runs
+
+
+def xlsx(path, doc, ocr_images=False, model=None):
+    import openpyxl
+    from openpyxl.utils import get_column_letter
+    values = openpyxl.load_workbook(path, data_only=True)
+    formulas = openpyxl.load_workbook(path, data_only=False)
+    props = values.properties
+    for k in ("title", "creator", "created", "modified", "lastModifiedBy"):
+        v = getattr(props, k, None)
+        if v:
+            doc["metadata"][k] = v.isoformat() if isinstance(v, dt.datetime) else str(v)
+    for ws, wf in zip(values.worksheets, formulas.worksheets):
+        info = {"name": ws.title, "state": ws.sheet_state, "dimensions": ws.dimensions,
+                "merged": [str(m) for m in ws.merged_cells.ranges], "formulas": {}, "uncalculated": 0}
+        grid = []
+        for row_v, row_f in zip(ws.iter_rows(), wf.iter_rows()):
+            out = []
+            for cv, cf in zip(row_v, row_f):
+                v = _cell_value(cv.value)
+                if isinstance(cf.value, str) and cf.value.startswith("="):
+                    info["formulas"][cf.coordinate] = cf.value
+                    if v is None:  # saved without cached results (e.g. written by a script)
+                        info["uncalculated"] += 1
+                        v = cf.value
+                out.append(v)
+            grid.append(out)
+        # Values of merged ranges live in the top-left cell; the rest are empty, which is correct for tables.
+        info["images"] = len(getattr(ws, "_images", []))
+        info["charts"] = len(getattr(ws, "_charts", []))
+        doc["sheets"].append(info)
+        doc["blocks"].append({"type": "heading", "level": 2, "text": ws.title, "sheet": ws.title, "source": "structure"})
+        for r0, c0, rows in _regions(grid):
+            a = f"{get_column_letter(c0 + 1)}{r0 + 1}"
+            b = f"{get_column_letter(c0 + len(rows[0]))}{r0 + len(rows)}"
+            doc["blocks"].append({"type": "table", "sheet": ws.title, "range": f"{a}:{b}", "source": "structure",
+                                  "rows": [["" if v is None else v for v in r] for r in rows]})
+        if ocr_images:
+            blobs = []
+            for im in getattr(ws, "_images", []):
+                try:
+                    blobs.append(im._data())
+                except Exception:
+                    pass
+            _ocr_blobs(blobs, doc, model)
+    doc["pages"] = len(values.worksheets)
+
+
+def delimited(path, doc):
+    raw = Path(path).read_bytes()
+    for enc in ("utf-8-sig", "utf-16", "cp1252", "latin-1"):
+        try:
+            text = raw.decode(enc)
+            break
+        except UnicodeDecodeError:
+            continue
+    try:
+        dialect = csv.Sniffer().sniff(text[:20000], delimiters=",;\t|")
+    except csv.Error:
+        dialect = csv.excel_tab if Path(path).suffix.lower() == ".tsv" else csv.excel
+    rows = [r for r in csv.reader(io.StringIO(text), dialect)]
+    doc["blocks"].append({"type": "table", "rows": rows, "source": "structure"})
+    doc["metadata"].update({"encoding": enc, "delimiter": dialect.delimiter, "rows": len(rows)})
+    doc["pages"] = 1
+
+
+def text_file(path, doc):
+    text = Path(path).read_text(encoding="utf-8", errors="replace")
+    for para in [p.strip() for p in text.split("\n\n") if p.strip()]:
+        if para.startswith("#"):
+            level = len(para) - len(para.lstrip("#"))
+            doc["blocks"].append({"type": "heading", "level": level, "text": para.lstrip("# ").strip(), "source": "structure"})
+        else:
+            doc["blocks"].append({"type": "paragraph", "text": para, "source": "structure"})
+    doc["pages"] = 1
+
+
+# ---------- output ----------
+
+def _block_text(b):
+    if b["type"] == "table":
+        return "\n".join("\t".join(str(c) for c in row) for row in b["rows"])
+    return b.get("text", "")
+
+
+def to_text(doc):
+    return "\n\n".join(plain(t) for t in (_block_text(b) for b in doc["blocks"]) if t)
+
+
+def _md_cell(v):
+    return str(v).replace("|", "\\|").replace("\n", " ")
+
+
+def _html_table(rows):
+    esc = lambda v: html.escape(str(v)).replace("\n", "<br>")  # noqa: E731
+    head, *body = rows
+    out = ["<table>", "<tr>" + "".join(f"<th>{esc(c)}</th>" for c in head) + "</tr>"]
+    out += ["<tr>" + "".join(f"<td>{esc(c)}</td>" for c in r) + "</tr>" for r in body]
+    return "\n".join(out + ["</table>"])
+
+
+def to_markdown(doc, tables="markdown", page_marks=True):
+    """tables="html" writes <table> elements, which keep cells containing pipes or line breaks intact."""
+    out, page = [], None
+    for b in doc["blocks"]:
+        if (page_marks and b.get("page") and doc["pages"] > 1 and b["page"] != page
+                and doc["format"] not in ("xlsx", "xlsm", "xltx")):
+            page = b["page"]
+            out.append(f"<!-- page {page} -->")
+        t = b["type"]
+        if t == "heading":
+            out.append("#" * min(6, max(1, b.get("level", 1))) + " " + b["text"])
+        elif t == "list_item":
+            text = re.sub(r"^(•|\*|-)\s+", "", b["text"])
+            prefix = "" if re.match(r"^\d{1,3}[.)]", text) else "1. " if b.get("ordered") else "- "
+            out.append("  " * b.get("level", 0) + prefix + text)
+        elif t == "table":
+            rows = b["rows"]
+            if not rows:
+                continue
+            n = max(len(r) for r in rows)
+            rows = [list(r) + [""] * (n - len(r)) for r in rows]
+            if tables == "html":
+                out.append(_html_table(rows))
+                continue
+            lines = ["| " + " | ".join(_md_cell(c) for c in rows[0]) + " |", "|" + " --- |" * n]
+            lines += ["| " + " | ".join(_md_cell(c) for c in r) + " |" for r in rows[1:]]
+            out.append("\n".join(lines))
+        elif t == "figure":
+            out.append("[figure]" + (f"\n\n> {b['text']}" if b.get("text") else ""))
+        elif t in ("header", "footer"):
+            out.append(f"<!-- {t}: {b['text']} -->")
+        else:
+            out.append(b["text"])
+    return "\n\n".join(out) + "\n"

docparse/extractbench.py

@@ -0,0 +1,120 @@
+"""ExtractBench (LlamaIndex) integration: a two-stage parse -> local LLM extraction pipeline.
+
+ExtractBench scores schema-guided extraction (document + JSON Schema -> JSON). A parser alone
+cannot fill a schema, so docparse takes part the way ExtractBench's own two-stage baselines do:
+it is the *parse stage*, and a language model reads its Markdown and returns the JSON.
+
+To isolate what the parser contributes, the same model, prompt, context budget and scorer are
+used with two parse stages:
+    docparse_extract_<model>  docparse Markdown (headings, HTML tables, OCR for scanned pages)
+    pypdf_extract_<model>     plain pypdf text, the simplest local baseline
+
+The model is served by any OpenAI-compatible endpoint (e.g. llama.cpp's server on a CPU):
+    python -m llama_cpp.server --model qwen2.5-3b-instruct-q4_k_m.gguf --n_ctx 12288 --port 8088
+    DOCPARSE_LLM_URL=http://127.0.0.1:8088 python scripts/extractbench_docparse.py run docparse_extract_local --test
+"""
+import json
+from pathlib import Path
+from typing import Any
+
+from extract_bench.extensions import register_pipeline, register_provider
+from extract_bench.inference.providers.extract.vllm_extract import VLLMExtractProvider
+from extract_bench.schemas.pipeline import PipelineSpec
+from extract_bench.schemas.product import ProductType
+
+from .extract import extract, to_markdown
+
+CHARS_PER_TOKEN = 2.5  # conservative: JSON schemas and numbers tokenise densely
+
+
+def docparse_pages(path):
+    doc = extract(path)
+    pages = []
+    for n in range(1, max(1, doc["pages"]) + 1):
+        blocks = [b for b in doc["blocks"] if b.get("page", 1) == n]
+        pages.append(to_markdown({**doc, "blocks": blocks, "pages": 1}, tables="html", page_marks=False).strip())
+    return pages
+
+
+def pypdf_pages(path):
+    from pypdf import PdfReader
+    return [(p.extract_text() or "").strip() for p in PdfReader(str(path)).pages]
+
+
+PARSERS = {"docparse": docparse_pages, "pypdf": pypdf_pages}
+
+
+@register_provider("parsed_text_local_llm")
+class ParsedTextLocalLLM(VLLMExtractProvider):
+    """ExtractBench's self-hosted provider, fed parsed page text instead of page images.
+
+    Extra config: parser ("docparse" | "pypdf"), n_ctx (served context length, tokens).
+    Everything else (JSON salvage, schema pruning, normalisation, scoring) is the harness's own.
+    """
+
+    def __init__(self, provider_name: str, base_config: dict[str, Any] | None = None):
+        super().__init__(provider_name, base_config)
+        self._parser = self.base_config.get("parser", "docparse")
+        self._n_ctx = int(self.base_config.get("n_ctx", 16384))
+
+    def _render_page_images(self, source_path: Path) -> list[str]:
+        return [f"<!-- Page {i} -->\n{t}" for i, t in enumerate(PARSERS[self._parser](source_path), 1)]
+
+    def _build_user_content(self, pages: list[str], schema: dict[str, Any], compact_schema: bool = False):
+        schema_text = json.dumps(schema, separators=(",", ":")) if compact_schema else json.dumps(schema, indent=2)
+        return ("Extract every field from the document text below according to the JSON schema.\n\n"
+                f"JSON schema:\n{schema_text}\n\nDocument text:\n" + "\n\n".join(pages) + "\n\n"
+                "Return only one valid JSON object matching the schema. Do not wrap it in markdown fences.\n\n"
+                f"{self._user_instruction}")
+
+    @staticmethod
+    def _fit_pages(pages, budget):
+        kept, used = [], 0
+        for p in pages:
+            if used + len(p) > budget:
+                if budget - used > 200:
+                    kept.append(p[: budget - used])
+                    used = budget
+                return kept, used, len(pages) - len(kept)
+            kept.append(p)
+            used += len(p) + 2
+        return kept, used, 0
+
+    def _call_api_fitting_context(self, schema: dict[str, Any], pages: list[str]) -> dict[str, Any]:
+        """Fit schema + text + output into the served context: compact the schema, prune its trailing
+        fields if it alone is too large, then cut the document text to the remaining budget.
+        Both parse stages get exactly the same budget."""
+        cur, pruned = schema, 0
+        out_tokens = min(self._max_tokens, self._n_ctx // 4)
+        overhead = len(self._system_prompt) + len(self._user_instruction) + 400
+        while True:
+            schema_chars = len(json.dumps(cur, separators=(",", ":")))
+            budget = int((self._n_ctx - out_tokens) * CHARS_PER_TOKEN) - schema_chars - overhead
+            if budget > 2000 or (nxt := self._prune_schema(cur)) is None:
+                break
+            cur, pruned = nxt, pruned + 1
+        for _ in range(4):
+            kept, used, cut = self._fit_pages(pages, budget)
+            try:
+                out = self._call_api(cur, kept, max_tokens=out_tokens, compact_schema=True)
+                break
+            except Exception as e:  # the server rejects requests over its context window: halve the text
+                if not any(k in str(e).lower() for k in ("context", "exceed", "400")):
+                    raise
+                budget //= 2
+        else:
+            raise RuntimeError("request does not fit the model context even with a small part of the text")
+        out["_config"].update(parser=self._parser, n_ctx=self._n_ctx, text_chars_sent=used)
+        if cut:
+            out["pages_truncated"], out["truncated"] = cut, True
+        if pruned:
+            out["schema_prune_rounds"], out["truncated"] = pruned, True
+        return out
+
+
+for parser in PARSERS:
+    register_pipeline(PipelineSpec(
+        pipeline_name=f"{parser}_extract_local", provider_name="parsed_text_local_llm",
+        product_type=ProductType.EXTRACT,
+        config={"parser": parser, "endpoint_env_var": "DOCPARSE_LLM_URL", "model": "qwen2.5-3b", "n_ctx": 12288,
+                "max_tokens": 4096, "structured_output": False, "temperature": 0.0, "timeout_s": 3600}))

docparse/ocr/charset.py

@@ -0,0 +1,38 @@
+"""Characters the OCR network can emit. Index 0 is the CTC blank."""
+import unicodedata
+
+CHARS = "".join(chr(c) for c in range(32, 127)) + "•€£°©"
+BLANK = 0
+INDEX = {c: i + 1 for i, c in enumerate(CHARS)}
+CLASSES = len(CHARS) + 1
+
+_MAP = {"‘": "'", "’": "'", "‚": "'", "“": '"', "”": '"', "„": '"',
+        "–": "-", "—": "-", "−": "-", "…": "...", " ": " ", "\t": " ",
+        "·": "•", "●": "•", "•": "•", "×": "x"}
+
+
+def normalize(text):
+    """Map text into CHARS: fold typographic punctuation and accents, drop the rest."""
+    out = []
+    for ch in text:
+        ch = _MAP.get(ch, ch)
+        if all(c in INDEX for c in ch):
+            out.append(ch)
+            continue
+        base = unicodedata.normalize("NFKD", ch).encode("ascii", "ignore").decode()
+        out.append(base if base and all(c in INDEX for c in base) else "")
+    return "".join(out)
+
+
+def encode(text):
+    return [INDEX[c] for c in text if c in INDEX]
+
+
+def decode_greedy(ids):
+    """Collapse repeats, drop blanks."""
+    out, prev = [], BLANK
+    for i in ids:
+        if i != prev and i != BLANK:
+            out.append(CHARS[i - 1])
+        prev = i
+    return "".join(out)

docparse/ocr/metrics.py

@@ -0,0 +1,29 @@
+"""Character and word error rates (edit distance / reference length)."""
+
+
+def edit_distance(a, b):
+    if len(a) < len(b):
+        a, b = b, a
+    prev = list(range(len(b) + 1))
+    for i, ca in enumerate(a, 1):
+        cur = [i]
+        for j, cb in enumerate(b, 1):
+            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
+        prev = cur
+    return prev[-1]
+
+
+def cer(pred, ref):
+    return edit_distance(pred, ref) / max(1, len(ref))
+
+
+def wer(pred, ref):
+    return edit_distance(pred.split(), ref.split()) / max(1, len(ref.split()))
+
+
+def summary(preds, refs):
+    chars = sum(edit_distance(p, r) for p, r in zip(preds, refs))
+    words = sum(edit_distance(p.split(), r.split()) for p, r in zip(preds, refs))
+    return {"lines": len(refs), "cer": round(chars / max(1, sum(len(r) for r in refs)), 5),
+            "wer": round(words / max(1, sum(len(r.split()) for r in refs)), 5),
+            "line_exact": round(sum(p == r for p, r in zip(preds, refs)) / max(1, len(refs)), 4)}

docparse/ocr/model.py

@@ -0,0 +1,77 @@
+"""A small CRNN line recognizer: convolutions read glyph shapes, a BiLSTM reads context, CTC aligns."""
+import numpy as np
+import torch
+from torch import nn
+
+from .charset import CLASSES, decode_greedy
+from .render import HEIGHT
+
+
+def _block(cin, cout, stride=1):
+    return [nn.Conv2d(cin, cout, 3, stride=stride, padding=1, bias=False), nn.BatchNorm2d(cout), nn.ReLU(inplace=True)]
+
+
+class LineOCR(nn.Module):
+    def __init__(self, width=32, hidden=128):
+        super().__init__()
+        self.config = {"width": width, "hidden": hidden}
+        w = width
+        self.cnn = nn.Sequential(
+            *_block(1, w, 2),                                      # 16 x W/2 (strided stem keeps CPU cost low)
+            *_block(w, 2 * w), nn.MaxPool2d(2),                    # 8 x W/4
+            *_block(2 * w, 4 * w), nn.MaxPool2d((2, 1)),           # 4 x W/4
+            *_block(4 * w, 4 * w), nn.MaxPool2d((2, 1)),           # 2 x W/4
+            *_block(4 * w, 8 * w),
+            nn.Conv2d(8 * w, 8 * w, (2, 1)), nn.BatchNorm2d(8 * w), nn.ReLU(inplace=True))  # 1 x W/4
+        self.rnn = nn.LSTM(8 * w, hidden, num_layers=2, bidirectional=True, batch_first=True, dropout=0.1)
+        self.head = nn.Linear(2 * hidden, CLASSES)
+
+    def forward(self, x):
+        """x: (B,1,HEIGHT,W) -> log-probs (B, W/4, CLASSES)."""
+        f = self.cnn(x).squeeze(2).transpose(1, 2)
+        f, _ = self.rnn(f)
+        return self.head(f).log_softmax(-1)
+
+
+def batch_tensor(images, pad_value=None):
+    """List of uint8 (HEIGHT, w) arrays -> float tensor (B,1,HEIGHT,maxW), dark text made positive."""
+    wmax = max(a.shape[1] for a in images)
+    wmax = (wmax + 3) // 4 * 4
+    out = np.zeros((len(images), 1, HEIGHT, wmax), np.float32)
+    for i, a in enumerate(images):
+        x = 1.0 - a.astype(np.float32) / 255.0
+        if x[:, :2].mean() + x[:, -2:].mean() > 1.0 and x.mean() > 0.5:  # light text on dark: flip to dark-on-light
+            x = 1.0 - x
+        bg = np.median(np.concatenate([x[0], x[-1], x[:, 0], x[:, -1]]))
+        x = x - bg  # background to zero, so right padding matches it
+        out[i, 0, :, :a.shape[1]] = x
+    return torch.from_numpy(out), [a.shape[1] // 4 for a in images]
+
+
+class Recognizer:
+    """Load trained weights and read line images."""
+
+    def __init__(self, path, device="cpu"):
+        data = torch.load(path, map_location=device, weights_only=True)
+        self.net = LineOCR(**data["config"]).to(device).eval()
+        self.net.load_state_dict(data["state"])
+        self.device = device
+        self.info = data.get("info", {})
+
+    @torch.inference_mode()
+    def read(self, images, batch=32):
+        """images: uint8 grayscale arrays of height HEIGHT -> [(text, confidence)]."""
+        results = [None] * len(images)
+        order = sorted(range(len(images)), key=lambda i: images[i].shape[1])
+        for s in range(0, len(order), batch):
+            idx = order[s:s + batch]
+            x, lens = batch_tensor([images[i] for i in idx])
+            logp = self.net(x.to(self.device)).cpu()
+            for k, i in enumerate(idx):
+                lp = logp[k, :max(1, lens[k])]
+                best = lp.argmax(-1)
+                text = decode_greedy(best.tolist())
+                emitted = best != 0
+                conf = float(lp.max(-1).values[emitted].exp().mean()) if emitted.any() else 0.0
+                results[i] = (text.strip(), round(conf, 4))
+        return results

docparse/ocr/page.py

@@ -0,0 +1,622 @@
+"""Page images -> text lines, ruled tables and layout blocks (classical image processing, numpy only).
+
+The neural part is only the line recognizer (model.py). Finding lines, columns and
+tables here is rule-based, so its failure modes are predictable:
+- ruled tables are read cell by cell; borderless tables are found from aligned gaps;
+- multi-column text is split on wide vertical gaps;
+- photos/figures are reported as `figure` blocks and not read.
+"""
+import re
+
+import numpy as np
+from PIL import Image
+
+from .render import HEIGHT
+
+LIST_RE = re.compile(r"^((•|-|\*|\d{1,3}[.)]|\(?[a-zA-Z][.)]|\([ivx]+\))\s+|\d{1,3}[.)](?=[A-Za-z]))")
+
+
+# ---------- small numpy morphology ----------
+
+def _runs(mask):
+    """Start/end (exclusive) of True runs in a 1-D boolean array."""
+    m = np.concatenate([[False], mask, [False]]).astype(np.int8)
+    d = np.diff(m)
+    return list(zip(np.flatnonzero(d == 1), np.flatnonzero(d == -1)))
+
+
+def _box_mean(a, r):
+    """Mean over a (2r+1)^2 window via an integral image."""
+    p = np.pad(a, r + 1, mode="edge").astype(np.float64)
+    s = p.cumsum(0).cumsum(1)
+    k = 2 * r + 1
+    tot = s[k:, k:] - s[:-k, k:] - s[k:, :-k] + s[:-k, :-k]
+    return (tot / (k * k))[:a.shape[0], :a.shape[1]]
+
+
+def _open_1d(mask, k, axis):
+    """Morphological opening with a 1 x k line: keeps only runs at least k long."""
+    m = mask if axis == 1 else mask.T
+    c = np.pad(m.astype(np.int32), ((0, 0), (1, 0))).cumsum(1)
+    full = (c[:, k:] - c[:, :-k]) == k          # window starting at j is all ink
+    out = np.zeros_like(m)
+    c2 = np.pad(full.astype(np.int32), ((0, 0), (k, 0))).cumsum(1)
+    cover = (c2[:, k:] - c2[:, :-k]) > 0       # any full window covers this pixel
+    out[:, :cover.shape[1]] = cover[:, :m.shape[1]]
+    return out if axis == 1 else out.T
+
+
+def components(ink):
+    """8-connected components from row runs. Returns rows, starts, ends and a list of run-index groups."""
+    H = ink.shape[0]
+    m = np.pad(ink.astype(np.int8), ((0, 0), (1, 1)))
+    d = np.diff(m, axis=1)
+    rows, starts = np.nonzero(d == 1)
+    _, ends = np.nonzero(d == -1)
+    n = len(rows)
+    parent = np.arange(n)
+
+    def find(i):
+        while parent[i] != i:
+            parent[i] = parent[parent[i]]
+            i = parent[i]
+        return i
+    row_start = np.searchsorted(rows, np.arange(H + 1))
+    for y in range(1, H):
+        i, a1 = row_start[y - 1], row_start[y]
+        j, b1 = row_start[y], row_start[y + 1]
+        while i < a1 and j < b1:  # runs in consecutive rows touch (diagonals included)
+            if starts[i] <= ends[j] and starts[j] <= ends[i]:
+                ri, rj = find(i), find(j)
+                if ri != rj:
+                    parent[ri] = rj
+            if ends[i] < ends[j]:
+                i += 1
+            else:
+                j += 1
+    roots = np.array([find(i) for i in range(n)], dtype=np.int64)
+    order = np.argsort(roots, kind="stable")
+    groups = np.split(order, np.flatnonzero(np.diff(roots[order])) + 1) if n else []
+    return rows, starts, ends, groups
+
+
+def drop_components(ink, test):
+    """Remove every component for which test(h, w, pixels) is true."""
+    rows, starts, ends, groups = components(ink)
+    out = ink.copy()
+    lengths = ends - starts
+    for grp in groups:
+        h = rows[grp].max() + 1 - rows[grp].min()
+        w = ends[grp].max() - starts[grp].min()
+        if test(h, w, int(lengths[grp].sum())):
+            for k in grp:
+                out[rows[k], starts[k]:ends[k]] = False
+    return out
+
+
+def remove_frames(ink, th):
+    """Drop outline shapes (rounded boxes, circles, slide decorations): components much larger than
+    a letter in both directions whose bounding box is mostly empty."""
+    return drop_components(ink, lambda h, w, px: h > 2.5 * th and w > 2.5 * th and px < 0.08 * h * w)
+
+
+# ---------- preprocessing ----------
+
+def load_gray(img):
+    if not isinstance(img, Image.Image):
+        img = Image.open(img)
+    if img.mode in ("RGBA", "LA", "P"):
+        img = img.convert("RGBA")
+        bg = Image.new("RGBA", img.size, (255, 255, 255, 255))
+        img = Image.alpha_composite(bg, img)
+    g = np.asarray(img.convert("L"), dtype=np.float32)
+    if np.median(g) < 110:  # dark mode / inverted scan
+        g = 255 - g
+    return g
+
+
+def binarize(g):
+    """Ink mask. Dark text on light paper, plus light text on dark fills (buttons, banners, headers).
+    Specks of scan noise (components of a few pixels) are removed."""
+    r = max(7, min(g.shape) // 60)
+    local = _box_mean(g, r)
+    gs, k = g, 12.0
+    if 1.4826 * float(np.median(np.abs(g - local))) > 6:  # grainy scan: noise level measured robustly (MAD)
+        gs = _box_mean(g, 1)  # 3x3 smoothing: grain averages out, strokes do not
+        k = max(12.0, 3.0 * 1.4826 * float(np.median(np.abs(gs - local))))
+    dark_bg = _box_mean((g < 150).astype(np.float32), r) > 0.55  # mostly dark surroundings (outside edges stay < 0.5)
+    ink = np.where(dark_bg, (gs > local + max(18.0, k)) & (gs > 120), (gs < local - k) & (gs < 215))
+    return drop_components(ink, lambda h, w, px: px <= 4)
+
+
+def _deskew_angle(ink):
+    small = Image.fromarray((ink * 255).astype(np.uint8))
+    scale = 900 / max(small.size)
+    if scale < 1:
+        small = small.resize((int(small.width * scale), int(small.height * scale)), Image.BILINEAR)
+    best, best_a = -1.0, 0.0
+    for a in np.arange(-4, 4.01, 0.25):
+        prof = np.asarray(small.rotate(a, resample=Image.BILINEAR), dtype=np.float32).sum(1)
+        score = float(np.square(np.diff(prof)).sum())
+        if score > best:
+            best, best_a = score, float(a)
+    return best_a
+
+
+def prepare(img, deskew=True):
+    """-> gray (float), ink mask, scale applied, deskew angle."""
+    g = load_gray(img)
+    scale = 1.0
+    if max(g.shape) < 1100:  # small screenshots: upsample so thin strokes survive binarization
+        scale = 2.0
+        g = np.asarray(Image.fromarray(g.astype(np.uint8)).resize((g.shape[1] * 2, g.shape[0] * 2), Image.BICUBIC),
+                       dtype=np.float32)
+    ink = binarize(g)
+    angle = 0.0
+    if deskew and ink.any():
+        angle = _deskew_angle(ink)
+        if abs(angle) >= 0.25:
+            g = np.asarray(Image.fromarray(g.astype(np.uint8)).rotate(angle, resample=Image.BICUBIC, fillcolor=255),
+                           dtype=np.float32)
+            ink = binarize(g)
+    return g, ink, scale, angle
+
+
+def text_height(ink):
+    """Typical text-line height. Row bands give it on plain pages; frames, logos and shapes merge
+    bands, so it is capped by the length of vertical letter strokes (stems ~ 0.7 x line height)."""
+    bands = [b - a for a, b in _runs(ink.sum(1) > max(1, ink.shape[1] * 0.002)) if b - a >= 4]
+    th = float(np.median(bands)) if bands else 20.0
+    sample = ink[:, ::3].T  # every third column is plenty
+    m = np.pad(sample.astype(np.int8), ((0, 0), (1, 1)))
+    d = np.diff(m, axis=1)
+    runs = np.flatnonzero(d.ravel() == -1) - np.flatnonzero(d.ravel() == 1)
+    runs = runs[(runs >= 5) & (runs < ink.shape[0] // 6)]
+    if len(runs) > 200:
+        th = min(th, 1.6 * float(np.percentile(runs, 90)))
+    return th
+
+
+# ---------- ruled tables ----------
+
+def _segments(mask, axis):
+    """Line pixels -> merged segments (pos0, pos1, start, end) along `axis` (1: horizontal lines)."""
+    segs = []
+    m = mask if axis == 1 else mask.T
+    for i in np.flatnonzero(m.any(1)):
+        for s, e in _runs(m[i]):
+            for seg in segs:
+                if seg[1] >= i - 1 and s < seg[3] and e > seg[2]:
+                    seg[1] = i
+                    seg[2], seg[3] = min(seg[2], s), max(seg[3], e)
+                    break
+            else:
+                segs.append([i, i, s, e])
+    return segs
+
+
+def find_tables(ink, th):
+    """Grid tables from long horizontal and vertical rules. Returns (tables, rule mask)."""
+    h = _open_1d(ink, int(max(40, 5 * th)), 1)
+    v = _open_1d(ink, int(max(20, 1.6 * th)), 0)
+    hs, vs = _segments(h, 1), _segments(v, 0)
+    # Union-find over rule segments that touch each other.
+    items = [("h", s) for s in hs] + [("v", s) for s in vs]
+    parent = list(range(len(items)))
+
+    def find(i):
+        while parent[i] != i:
+            parent[i] = parent[parent[i]]
+            i = parent[i]
+        return i
+    tol = 4
+    for i, (ki, a) in enumerate(items):
+        if ki != "h":
+            continue
+        for j, (kj, b) in enumerate(items):
+            if kj != "v":
+                continue
+            # h: rows a0..a1, cols a2..a3 ; v: cols b0..b1, rows b2..b3
+            if b[0] - tol <= a[3] and b[1] + tol >= a[2] and a[0] + tol >= b[2] and a[1] - tol <= b[3]:
+                parent[find(i)] = find(j)
+    groups = {}
+    for i in range(len(items)):
+        groups.setdefault(find(i), []).append(items[i])
+    tables = []
+    for g in groups.values():
+        H = [s for k, s in g if k == "h"]
+        V = [s for k, s in g if k == "v"]
+        ys = _cluster([(s[0] + s[1]) / 2 for s in H])
+        xs = _cluster([(s[0] + s[1]) / 2 for s in V])
+        if len(ys) < 2 or len(xs) < 2 or (len(ys) == 2 and len(xs) == 2):
+            continue  # a plain box or single rule is not a table
+        tables.append({"ys": ys, "xs": xs, "H": H, "V": V,
+                       "bbox": [min(xs), min(ys), max(xs), max(ys)]})
+    return tables, h | v
+
+
+def _cluster(vals, tol=6):
+    out = []
+    for v in sorted(vals):
+        if out and v - out[-1][-1] <= tol:
+            out[-1].append(v)
+        else:
+            out.append([v])
+    return [float(np.mean(c)) for c in out]
+
+
+def _covered(segs, pos, a, b, axis_pos, tol=4):
+    """Fraction of [a,b] covered by rules lying at `pos` (rules given as (p0,p1,s,e))."""
+    cov = np.zeros(int(b - a) + 1, bool)
+    for s in segs:
+        if s[0] - tol <= pos <= s[1] + tol:
+            lo, hi = max(a, s[2]), min(b, s[3])
+            if hi > lo:
+                cov[int(lo - a):int(hi - a)] = True
+    return cov.mean() if len(cov) else 0.0
+
+
+# ---------- text lines ----------
+
+def _trim(ink, y0, y1, x0, x1):
+    sub = ink[y0:y1, x0:x1]
+    rows, cols = np.flatnonzero(sub.any(1)), np.flatnonzero(sub.any(0))
+    if not len(rows):
+        return None
+    return y0 + rows[0], y0 + rows[-1] + 1, x0 + cols[0], x0 + cols[-1] + 1
+
+
+def _split_tall(ink, box, th):
+    """Split a band that holds several touching lines at deep minima of its ink profile."""
+    y0, y1, x0, x1 = box
+    if y1 - y0 < 1.7 * th:
+        return [box]
+    prof = ink[y0:y1, x0:x1].sum(1).astype(float)
+    lo, hi = int(0.6 * th), int(y1 - y0 - 0.6 * th)
+    if hi <= lo:
+        return [box]
+    k = lo + int(np.argmin(prof[lo:hi]))
+    if prof[k] > 0.15 * np.median(prof[prof > 0]):
+        return [box]
+    a, b = _trim(ink, y0, y0 + k, x0, x1), _trim(ink, y0 + k, y1, x0, x1)
+    return (_split_tall(ink, a, th) if a else []) + (_split_tall(ink, b, th) if b else [])
+
+
+def xy_cut(ink, box, th, out):
+    """Recursive XY-cut. Appends text-line boxes (y0,y1,x0,x1) to `out` in reading order.
+
+    Order of preference: unusually large row gaps (section breaks), then a column gutter
+    that runs through the whole region, then ordinary line gaps.
+    """
+    t = _trim(ink, *box)
+    if t is None:
+        return
+    y0, y1, x0, x1 = t
+    sub = ink[y0:y1, x0:x1]
+    row_gaps = _runs(~sub.any(1))
+    if len(row_gaps) >= 2:
+        typical = float(np.median([b - a for a, b in row_gaps]))
+        big = [(a, b) for a, b in row_gaps if b - a >= max(0.8 * th, 1.6 * typical)]
+        if big and len(big) < len(row_gaps):
+            cuts = [y0] + [y0 + (a + b) // 2 for a, b in big] + [y1]
+            for a, b in zip(cuts, cuts[1:]):
+                xy_cut(ink, (a, b, x0, x1), th, out)
+            return
+    gutter = _column_gutter(sub, th)
+    if gutter:
+        a, b = gutter
+        xy_cut(ink, (y0, y1, x0, x0 + a), th, out)
+        xy_cut(ink, (y0, y1, x0 + b, x1), th, out)
+        return
+    if not row_gaps and y1 - y0 > 1.7 * th and len(_split_tall(ink, t, th)) > 1:
+        # Tall region with no blank row: something (a frame's curved end, an icon) spans every row.
+        # Cut it apart on blank columns, then the pieces can split into lines again.
+        cols = [(a, b) for a, b in _runs(~sub.any(0)) if b - a >= 0.5 * th]
+        if cols:
+            edges = [0] + [p for a, b in cols for p in (a, b)] + [sub.shape[1]]
+            for a, b in zip(edges[::2], edges[1::2]):
+                xy_cut(ink, (y0, y1, x0 + a, x0 + b), th, out)
+            return
+    if len(row_gaps) == 1 and row_gaps[0][1] - row_gaps[0][0] >= 0.8 * th:
+        a, b = row_gaps[0]
+        xy_cut(ink, (y0, y0 + (a + b) // 2, x0, x1), th, out)
+        xy_cut(ink, (y0 + (a + b) // 2, y1, x0, x1), th, out)
+        return
+    cuts = [0] + [(a + b) // 2 for a, b in row_gaps] + [sub.shape[0]]
+    for a, b in zip(cuts, cuts[1:]):
+        line = _trim(ink, y0 + a, y0 + b, x0, x1)
+        if line and line[1] - line[0] >= 3:
+            out.extend(_split_tall(ink, line, th))
+
+
+def _column_gutter(sub, th):
+    """A blank vertical strip through the whole region separating two blocks of running text."""
+    H, W = sub.shape
+    if H < 1.8 * th:
+        return None
+    gaps = [(a, b) for a, b in _runs(~sub.any(0)) if b - a >= 1.5 * th and a >= 0.2 * W and W - b >= 0.2 * W]
+    for a, b in sorted(gaps, key=lambda g: g[0] - g[1]):
+        left, right = sub[:, :a], sub[:, b:]
+        n = len(_runs(left.any(1))) + len(_runs(right.any(1)))
+        if n >= 4 and _is_text_column(left, th) and _is_text_column(right, th):
+            return a, b
+    return None
+
+
+def _is_text_column(side, th):
+    """Running text: several lines, wide enough for a phrase, and most lines span most of the width."""
+    W = side.shape[1]
+    if W < 10 * th:
+        return False
+    rows = [r for r in _runs(side.any(1)) if r[1] - r[0] >= 0.3 * th]
+    if not rows:
+        return False
+    fills = []
+    for a, b in rows:
+        cols = np.flatnonzero(side[a:b].any(0))
+        fills.append((cols[-1] - cols[0] + 1) / W)
+    if len(rows) <= 2:  # a short column: its first line must run the full width (wrapped prose)
+        return fills[0] >= 0.85
+    return float(np.median(fills)) >= 0.7
+
+
+def line_segments(ink, box, th, gap=1.4):
+    """Split a text line on wide horizontal gaps (table cells, tab stops)."""
+    y0, y1, x0, x1 = box
+    cols = ink[y0:y1, x0:x1].any(0)
+    segs, start = [], None
+    gaps = [(a, b) for a, b in _runs(~cols) if b - a >= gap * th]
+    edges = [0] + [p for a, b in gaps for p in (a, b)] + [len(cols)]
+    for s, e in zip(edges[::2], edges[1::2]):
+        t = _trim(ink, y0, y1, x0 + s, x0 + e)
+        if t:
+            segs.append(t)
+    return segs or [box]
+
+
+def crop(gray, box):
+    """Gray crop with a little margin, resized to the recognizer's input height."""
+    y0, y1, x0, x1 = box
+    h = y1 - y0
+    py, px = max(2, int(0.15 * h)), max(3, int(0.3 * h))
+    a = gray[max(0, y0 - py):min(gray.shape[0], y1 + py), max(0, x0 - px):min(gray.shape[1], x1 + px)]
+    border = np.median(np.concatenate([a[0], a[-1], a[:, 0], a[:, -1]]))
+    if border < a.mean():  # light text on a dark fill: make it dark-on-light like the training lines
+        a, border = 255 - a, 255 - border
+    lo = np.percentile(a, 2)
+    if border - lo > 20:  # stretch so the background is white and the strokes are dark
+        a = (a - lo) * (230.0 / (border - lo)) + 10
+    w = max(8, min(4000, round(a.shape[1] * HEIGHT / max(1, a.shape[0]))))
+    return np.asarray(Image.fromarray(np.clip(a, 0, 255).astype(np.uint8)).resize((w, HEIGHT), Image.BILINEAR))
+
+
+# ---------- whole page ----------
+
+def text_blocks(text_ink, th, read, figures=True):
+    """Text mask -> grouped blocks. read(boxes) -> [(text, confidence)] supplies the words for each box."""
+    lines = []
+    xy_cut(text_ink, (0, text_ink.shape[0], 0, text_ink.shape[1]), th, lines)
+    lines = [ln for ln in lines if (ln[1] - ln[0]) >= 0.35 * th or (ln[3] - ln[2]) >= 0.35 * th]
+    if lines:
+        th = float(np.median([ln[1] - ln[0] for ln in lines]))
+    figs, text_lines = [], []
+    for ln in lines:
+        h, w = ln[1] - ln[0], ln[3] - ln[2]
+        if figures and h > 4 * th and (text_ink[ln[0]:ln[1], ln[2]:ln[3]].mean() > 0.35 or w < 2 * h):
+            figs.append({"kind": "figure", "box": ln})
+        else:
+            text_lines.append(ln)
+    segs = [line_segments(text_ink, ln, th) for ln in text_lines]
+    flat = [s for ss in segs for s in ss]
+    got = read(flat) if flat else []
+    blocks, k = [], 0
+    for ln, ss in zip(text_lines, segs):
+        blocks.append({"kind": "line", "box": ln, "segs": ss, "parts": got[k:k + len(ss)],
+                       "xh": core_height(text_ink, ln)})
+        k += len(ss)
+    return blocks + figs, th
+
+
+def core_height(ink, box):
+    """Rows holding most of a line's ink (about the x-height): tracks font size better than the
+    full ink box, which depends on whether a line happens to contain ascenders and descenders."""
+    y0, y1, x0, x1 = box
+    prof = ink[y0:y1, x0:x1].sum(1)
+    return int((prof >= 0.4 * prof.max()).sum()) if prof.size and prof.max() > 0 else y1 - y0
+
+
+def analyze(img, recognizer, deskew=True):
+    """Image -> list of blocks: heading / paragraph / list_item / table / figure (bbox in original pixels)."""
+    gray, ink, scale, angle = prepare(img, deskew)
+    th = text_height(ink)
+    tables, rules = find_tables(ink, th)
+    H, W = ink.shape
+    frames = _open_1d(ink, max(40, W // 8), 1) | _open_1d(ink, max(40, H // 8), 0)  # boxes, banners, page rules
+    clean = remove_frames(ink & ~rules & ~frames, th)
+    text_ink = clean.copy()
+    for t in tables:  # table interiors are read cell by cell, not as running text
+        x0, y0, x1, y1 = (int(v) for v in t["bbox"])
+        text_ink[max(0, y0 - 2):y1 + 3, max(0, x0 - 2):x1 + 3] = False
+
+    def ocr(boxes):
+        return recognizer.read([crop(gray, b) for b in boxes])
+    th = min(th, text_height(text_ink))  # re-estimate without frames and tables
+    blocks, th = text_blocks(text_ink, th, ocr)
+    for t in tables:
+        blocks.append(read_grid(t, clean, th, ocr))
+    out = group(blocks, th)
+    for b in out:
+        y0, y1, x0, x1 = b.pop("box")
+        b["bbox"] = [round(x0 / scale), round(y0 / scale), round(x1 / scale), round(y1 / scale)]
+    meta = {"deskew_degrees": angle, "upscaled": scale, "text_height_px": round(th / scale, 1)}
+    return out, meta
+
+
+def read_grid(t, clean, th, read):
+    nr, nc = len(t["ys"]) - 1, len(t["xs"]) - 1
+    cells, jobs = [], []
+    for r in range(nr):
+        for c in range(nc):
+            y0, y1 = int(t["ys"][r]) + 3, int(t["ys"][r + 1]) - 2
+            x0, x1 = int(t["xs"][c]) + 3, int(t["xs"][c + 1]) - 2
+            sub = []
+            if y1 > y0 and x1 > x0:
+                xy_cut(clean, (y0, y1, x0, x1), th, sub)
+            cells.append((r, c, len(jobs), len(sub)))
+            jobs += sub
+    got = read(jobs) if jobs else []
+    grid = [["" for _ in range(nc)] for _ in range(nr)]
+    for r, c, k, n in cells:
+        grid[r][c] = " ".join(g[0] for g in got[k:k + n] if g[0])
+    _apply_spans(t, grid)
+    confs = [g[1] for g in got if g[1] is not None]
+    x0, y0, x1, y1 = t["bbox"]
+    return {"kind": "table", "box": (int(y0), int(y1), int(x0), int(x1)), "rows": grid,
+            "confidence": round(float(np.mean(confs)), 4) if confs else None}
+
+
+def _apply_spans(t, grid):
+    """Cells whose dividing rule is missing are merged: the text stays in the first cell."""
+    ys, xs = t["ys"], t["xs"]
+    for r in range(len(grid)):
+        for c in range(1, len(grid[0])):
+            if _covered(t["V"], xs[c], ys[r] + 2, ys[r + 1] - 2, 0) < 0.5 and not grid[r][c]:
+                grid[r][c] = None  # continuation of a merged cell
+    for r in range(1, len(grid)):
+        for c in range(len(grid[0])):
+            if _covered(t["H"], ys[r], xs[c] + 2, xs[c + 1] - 2, 1) < 0.5 and not grid[r][c]:
+                grid[r][c] = None
+    for row in grid:
+        for c, v in enumerate(row):
+            if v is None:
+                row[c] = ""
+
+
+def group(blocks, th):
+    """Lines -> headings, paragraphs, list items and borderless tables, keeping reading order.
+
+    Lines arrive in XY-cut order (columns already respected); tables and figures are
+    inserted before the first text block that starts below them.
+    """
+    lines = [b for b in blocks if b["kind"] == "line"]
+    gaps = [b["box"][0] - a["box"][1] for a, b in zip(lines, lines[1:])]
+    gaps = [g for g in gaps if 0 <= g < 2.5 * th]
+    lead = float(np.median(gaps)) if gaps else 0.5 * th  # usual space between lines of one paragraph
+    xh = float(np.median([b["xh"] for b in lines])) if lines else th
+    others = [b for b in blocks if b["kind"] != "line"]
+    out, i = [], 0
+    while i < len(lines):
+        ln = lines[i]
+        # Borderless table: a run of lines that each split into 2+ aligned segments.
+        j = i
+        while j < len(lines) and len(lines[j]["segs"]) >= 2:
+            j += 1
+        if j - i >= 2:
+            table = _borderless(lines[i:j])
+            if table:
+                out.append(table)
+                i = j
+                continue
+            for ln2 in lines[i:j]:  # not aligned like a table: keep as text lines
+                ln2["segs"], ln2["parts"] = [ln2["box"]], [(" ".join(p[0] for p in ln2["parts"] if p[0]),
+                                                            _mean([p[1] for p in ln2["parts"]]))]
+            # fall through and handle line i as text
+        text = " ".join(p[0] for p in ln["parts"] if p[0])
+        conf = [p[1] for p in ln["parts"]]
+        if not text or (len(text) <= 2 and (_mean(conf) or 1) < 0.6):
+            i += 1  # specks, frame corners, icons read as one or two unsure characters
+            continue
+        h = ln["box"][1] - ln["box"][0]
+        size = ln["xh"] / max(1.0, xh)  # font size relative to body text
+        prev = out[-1] if out else None
+        gap = ln["box"][0] - prev["box"][1] if prev else 1e9
+        big = size >= 1.22 or (size >= 1.1 and h >= 1.3 * th)
+        if big and len(text) < 120:
+            level = 1 if size >= 1.7 else 2 if size >= 1.32 else 3
+            if prev and prev["type"] == "heading" and prev["level"] == level and 0 <= gap < 1.0 * h:
+                _extend(prev, ln, text, conf)
+            else:
+                out.append({"type": "heading", "level": level, "text": text, "box": ln["box"], "conf": conf})
+        elif LIST_RE.match(text):
+            out.append({"type": "list_item", "text": text, "box": ln["box"], "conf": conf})
+        elif (prev and prev["type"] in ("paragraph", "list_item") and -0.5 * th < gap < max(0.9 * th, 1.35 * lead)
+              and abs(ln["box"][2] - prev["box"][2]) < 2.5 * th and not big):
+            _extend(prev, ln, text, conf)
+        else:
+            out.append({"type": "paragraph", "text": text, "box": ln["box"], "conf": conf})
+        i += 1
+    for o in others:
+        if o["kind"] == "figure":
+            o = {"type": "figure", "box": o["box"], **({"text": o["text"]} if o.get("text") else {})}
+        else:
+            o = {"type": "table", "rows": o["rows"], "box": o["box"], "confidence": o["confidence"]}
+        k = next((n for n, b in enumerate(out) if b["box"][0] > o["box"][0] and b["box"][2] < o["box"][3]), len(out))
+        out.insert(k, o)
+    for b in out:
+        if "lines" in b:
+            b["text"] = _join_lines(b.pop("lines"))
+        if "conf" in b:
+            c = [v for v in b.pop("conf") if v is not None]
+            b["confidence"] = round(float(np.mean(c)), 4) if c else None
+    return out
+
+
+def _extend(block, ln, text, conf):
+    y0, y1, x0, x1 = block["box"]
+    b = ln["box"]
+    block["box"] = (min(y0, b[0]), max(y1, b[1]), min(x0, b[2]), max(x1, b[3]))
+    block.setdefault("lines", [(block["text"], y1 - y0 and x1 - x0)]).append((text, b[3] - b[2]))
+    block["conf"] += conf
+
+
+def _join_lines(lines):
+    """Join wrapped lines with spaces but keep real line breaks (code, addresses, poems).
+
+    A wrapped line ends only because the next word did not fit; if the next line's first
+    word would have fitted after this line, the break was deliberate.
+    """
+    widest = max(w for _, w in lines)
+    out = lines[0][0]
+    for (prev, w), (text, _) in zip(lines, lines[1:]):
+        per_char = w / max(1, len(prev))
+        first = text.split()[0] if text.split() else ""
+        if w + per_char * (len(first) + 1) < 0.97 * widest:
+            out += "\n" + text
+        elif prev[-2:-1].islower() and prev.endswith("-") and text[:1].islower():
+            out = out[:-1] + text  # rejoin a hyphenated word
+        else:
+            out += " " + text
+    return out
+
+
+def _mean(vals):
+    vals = [v for v in vals if v is not None]
+    return round(float(np.mean(vals)), 4) if vals else None
+
+
+def _borderless(lines):
+    """Rows of aligned segments -> a table, or None when the segments do not line up in columns."""
+    x0 = min(ln["box"][2] for ln in lines)
+    x1 = max(ln["box"][3] for ln in lines)
+    occupied = np.zeros(x1 - x0 + 1, bool)
+    for ln in lines:
+        for s in ln["segs"]:
+            occupied[s[2] - x0:s[3] - x0] = True
+    spans = _runs(occupied)
+    if len(spans) < 2:
+        return None
+    rows, conf, fill = [], [], []
+    for ln in lines:
+        row, hit = [""] * len(spans), [False] * len(spans)
+        for s, (t, c) in zip(ln["segs"], ln["parts"]):
+            mid = (s[2] + s[3]) / 2 - x0
+            k = next((n for n, (a, b) in enumerate(spans) if a <= mid <= b), len(spans) - 1)
+            row[k] = (row[k] + " " + t).strip()
+            hit[k] = True
+            conf.append(c)
+        fill.append(sum(hit) / len(spans))
+        rows.append(row)
+    if np.mean(fill) < 0.6:  # e.g. code with alignment spaces: many sparse "columns"
+        return None
+    box = (min(ln["box"][0] for ln in lines), max(ln["box"][1] for ln in lines), x0, x1)
+    return {"type": "table", "rows": rows, "box": box, "conf": conf}

docparse/ocr/render.py

@@ -0,0 +1,214 @@
+"""Synthetic text-line images for training and testing the OCR network.
+
+Every line is drawn with a real font, then degraded the way scans, photos and
+screenshots degrade text. Fonts and text are split: the test side uses fonts and
+sentences that never appear in training (Carlito/Caladea are metric clones of
+Calibri/Cambria, the default Office fonts).
+"""
+import io
+import random
+from functools import lru_cache
+from pathlib import Path
+
+import numpy as np
+from PIL import Image, ImageDraw, ImageFilter, ImageFont
+
+from .charset import CHARS, normalize
+
+HEIGHT = 32
+MAX_WIDTH = 1200
+DATA = Path(__file__).resolve().parents[1] / "data"
+FONT_DIR = Path("/usr/share/fonts")
+
+TRAIN_FONTS = [
+    "truetype/dejavu/DejaVuSans.ttf", "truetype/dejavu/DejaVuSans-Bold.ttf", "truetype/dejavu/DejaVuSerif.ttf",
+    "truetype/dejavu/DejaVuSerif-Bold.ttf", "truetype/dejavu/DejaVuSansMono.ttf",
+    "truetype/liberation/LiberationSans-Regular.ttf", "truetype/liberation/LiberationSans-Bold.ttf",
+    "truetype/liberation/LiberationSans-Italic.ttf", "truetype/liberation/LiberationSerif-Regular.ttf",
+    "truetype/liberation/LiberationSerif-Bold.ttf", "truetype/liberation/LiberationSerif-Italic.ttf",
+    "truetype/liberation/LiberationMono-Regular.ttf", "truetype/freefont/FreeSans.ttf",
+    "truetype/freefont/FreeSansBold.ttf", "truetype/freefont/FreeSansOblique.ttf", "truetype/freefont/FreeSerif.ttf",
+    "truetype/freefont/FreeSerifBold.ttf", "truetype/freefont/FreeSerifItalic.ttf", "truetype/freefont/FreeMono.ttf",
+    "opentype/inter/Inter-Regular.otf", "opentype/inter/Inter-Bold.otf", "opentype/inter/Inter-Light.otf",
+    "opentype/inter/Inter-Medium.otf", "opentype/inter/Inter-Italic.otf", "opentype/inter/Inter-SemiBold.otf",
+    "truetype/jetbrains-mono/JetBrainsMono-Regular.ttf", "truetype/jetbrains-mono/JetBrainsMono-Bold.ttf",
+]
+TEST_FONTS = [
+    "truetype/crosextra/Carlito-Regular.ttf", "truetype/crosextra/Carlito-Bold.ttf",
+    "truetype/crosextra/Carlito-Italic.ttf", "truetype/crosextra/Caladea-Regular.ttf",
+    "truetype/crosextra/Caladea-Bold.ttf", "truetype/crosextra/Caladea-Italic.ttf",
+]
+
+
+def available(fonts):
+    return [str(FONT_DIR / f) for f in fonts if (FONT_DIR / f).exists()]
+
+
+@lru_cache(maxsize=4096)
+def _font(path, size):
+    return ImageFont.truetype(path, size)
+
+
+@lru_cache(maxsize=2)
+def corpus(split):
+    lines = (DATA / f"corpus_{split}.txt").read_text(encoding="utf-8").splitlines()
+    return [ln for ln in lines if ln.strip()]
+
+
+# ---------- text sampling ----------
+
+MONTHS = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October",
+          "November", "December"]
+WORDS = ["Invoice", "Total", "Subtotal", "Tax", "Amount", "Date", "Name", "Address", "Phone", "Email", "Qty",
+         "Price", "Description", "Item", "Balance", "Due", "Account", "Reference", "Page", "Table", "Figure",
+         "Summary", "Introduction", "Results", "Method", "Revenue", "Q1", "Q2", "Q3", "Q4", "Report", "ID",
+         "Status", "Approved", "Pending", "Customer", "Order", "Units", "Region", "North", "South", "Notes"]
+
+
+def _number(r):
+    k = r.random()
+    if k < 0.3:
+        return f"{r.randint(0, 999999):,}"
+    if k < 0.55:
+        return f"{r.uniform(-9999, 99999):,.2f}"
+    if k < 0.7:
+        return f"{r.uniform(0, 100):.1f}%"
+    if k < 0.85:
+        return r.choice("$€£") + f"{r.uniform(0, 50000):,.2f}"
+    return str(r.randint(0, 10 ** r.randint(1, 9)))
+
+
+def _structured(r):
+    k = r.randint(0, 11)
+    if k == 0:
+        return _number(r)
+    if k == 1:
+        y, m, d = r.randint(1990, 2030), r.randint(1, 12), r.randint(1, 28)
+        return r.choice([f"{y}-{m:02d}-{d:02d}", f"{d:02d}/{m:02d}/{y}", f"{MONTHS[m - 1]} {d}, {y}",
+                         f"{d} {MONTHS[m - 1][:3]} {y}"])
+    if k == 2:
+        user = "".join(r.choice("abcdefghijklmnopqrstuvwxyz._0123456789") for _ in range(r.randint(3, 12)))
+        return f"{user}@{r.choice(['gmail.com', 'example.org', 'mail.co.uk', 'corp.io'])}"
+    if k == 3:
+        path = "/".join("".join(r.choice("abcdefghijklmnopqrstuvwxyz-") for _ in range(r.randint(2, 8)))
+                        for _ in range(r.randint(1, 3)))
+        return f"https://www.{r.choice(['example', 'docs', 'data'])}.com/{path}"
+    if k == 4:
+        return f"{r.choice(['INV', 'PO', 'ID', 'REF', 'SKU', 'No.'])}-{r.randint(0, 99999):05d}"
+    if k == 5:
+        return f"+{r.randint(1, 99)} {r.randint(100, 999)} {r.randint(100, 999)} {r.randint(1000, 9999)}"
+    if k == 6:
+        return f"{r.choice(WORDS)}: {_number(r)}"
+    if k == 7:
+        return " ".join(_number(r) for _ in range(r.randint(2, 5)))
+    if k == 8:
+        return r.choice(["• ", "- ", "* ", f"{r.randint(1, 20)}. ", f"({r.choice('abcdef')}) "]) + _prose(r, 4, 40)
+    if k == 9:
+        return _prose(r, 3, 40).upper()
+    if k == 10:
+        return " ".join(w.capitalize() for w in _prose(r, 6, 45).split())
+    return "".join(r.choice(CHARS[:95]) for _ in range(r.randint(3, 25))).strip() or "x"
+
+
+def _prose(r, lo, hi, split="train"):
+    para = r.choice(corpus(split))
+    n = r.randint(lo, hi)
+    if len(para) <= n:
+        return para
+    i = r.randint(0, len(para) - n)
+    j = para.rfind(" ", 0, i) + 1  # start on a word boundary
+    end = para.find(" ", j + n)
+    return para[j:end if end > 0 else None].strip() or para[:n]
+
+
+def sample_text(r, split="train"):
+    k = r.random()
+    if k < 0.62:
+        t = _prose(r, 3, 70, split)
+    elif k < 0.95:
+        t = _structured(r)
+    else:
+        t = r.choice(WORDS)
+    return normalize(t).strip() or "a"
+
+
+# ---------- rendering ----------
+
+def render_line(text, font_path, size, r, degrade=True):
+    """Draw one line of text and return a uint8 grayscale array of height HEIGHT."""
+    font = _font(font_path, size)
+    ink = r.randint(0, 70)
+    bg = r.randint(200, 255)
+    invert = degrade and r.random() < 0.04
+    if invert:
+        ink, bg = r.randint(225, 255), r.randint(20, 90)
+    left, top, right, bottom = font.getbbox(text)
+    asc, desc = font.getmetrics()
+    pad = int(size * 0.6) + 8
+    w, h = right - left + 2 * pad, asc + desc + 2 * pad
+    img = Image.new("L", (w, h), bg)
+    d = ImageDraw.Draw(img)
+    d.text((pad - left, pad), text, font=font, fill=ink)
+    if degrade and r.random() < 0.06:  # underline
+        y = pad + asc + max(1, desc // 3)
+        d.line([pad, y, w - pad, y], fill=ink, width=max(1, size // 16))
+    if degrade and r.random() < 0.5:
+        img = img.rotate(r.uniform(-1.2, 1.2), resample=Image.BILINEAR, fillcolor=bg, expand=False)
+
+    # Crop: either the tight ink box or the font's full line box, plus random padding.
+    if r.random() < 0.5:
+        y0, y1 = pad + top, pad + bottom
+    else:
+        y0, y1 = pad, pad + asc + desc
+    x0, x1 = pad, pad + right - left
+    ph = (y1 - y0)
+    py0, py1 = (r.uniform(0.02, 0.25) * ph, r.uniform(0.02, 0.25) * ph) if degrade else (0.1 * ph, 0.1 * ph)
+    px0, px1 = (r.uniform(0.0, 0.6) * ph, r.uniform(0.0, 0.6) * ph) if degrade else (0.2 * ph, 0.2 * ph)
+    img = img.crop((int(x0 - px0), int(y0 - py0), int(x1 + px1) + 1, int(y1 + py1) + 1))
+
+    if degrade:
+        dr = ImageDraw.Draw(img)
+        if r.random() < 0.08:  # remnants of table rules at the crop edge
+            side = r.randint(0, 3)
+            W, H = img.size
+            lw = r.randint(1, 2)
+            [lambda: dr.line([0, 0, W, 0], fill=ink, width=lw), lambda: dr.line([0, H - 1, W, H - 1], fill=ink, width=lw),
+             lambda: dr.line([0, 0, 0, H], fill=ink, width=lw), lambda: dr.line([W - 1, 0, W - 1, H], fill=ink, width=lw)][side]()
+        if r.random() < 0.35:
+            scale = r.uniform(0.45, 0.85)  # resolution loss: low-dpi scans, small screenshots
+            small = img.resize((max(4, int(img.width * scale)), max(4, int(img.height * scale))), Image.BILINEAR)
+            img = small.resize(img.size, Image.BILINEAR)
+        elif r.random() < 0.3:
+            img = img.filter(ImageFilter.GaussianBlur(r.uniform(0.3, 0.9)))
+    out_w = max(8, min(MAX_WIDTH, round(img.width * HEIGHT / img.height)))
+    img = img.resize((out_w, HEIGHT), Image.BILINEAR)
+    a = np.asarray(img, dtype=np.float32)
+    if degrade:
+        if r.random() < 0.4:
+            a = a + np.random.default_rng(r.randint(0, 2 ** 31)).normal(0, r.uniform(2, 14), a.shape)
+        if r.random() < 0.15:  # uneven lighting
+            a = a * np.linspace(r.uniform(0.75, 1.0), r.uniform(0.75, 1.0), a.shape[1])[None, :]
+    a = np.clip(a, 0, 255).astype(np.uint8)
+    if degrade and r.random() < 0.25:
+        buf = io.BytesIO()
+        Image.fromarray(a).save(buf, "JPEG", quality=r.randint(25, 85))
+        a = np.asarray(Image.open(buf).convert("L"))
+    return a
+
+
+def sample(seed, split="train", degrade=True):
+    """One (image, text) pair. split chooses both the font list and the corpus."""
+    r = random.Random(seed)
+    fonts = available(TRAIN_FONTS if split == "train" else TEST_FONTS)
+    text = sample_text(r, split)
+    size = r.randint(14, 44)
+    return render_line(text, r.choice(fonts), size, r, degrade), text
+
+
+def dataset(count, seed, split="train", degrade=True):
+    images, texts = [], []
+    for i in range(count):
+        a, t = sample(seed * 1_000_003 + i, split, degrade)
+        images.append(a)
+        texts.append(t)
+    return images, texts

docparse/ocr/train.py

@@ -0,0 +1,108 @@
+"""Train the line recognizer from scratch on freshly rendered synthetic lines.
+
+Each round renders a new set of lines (new seeds), so the network never sees the
+same image twice. A checkpoint is written after every round; --resume continues.
+"""
+import json
+import time
+from multiprocessing import Pool
+from pathlib import Path
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+
+from .charset import encode
+from .metrics import summary
+from .model import LineOCR, Recognizer, batch_tensor
+from .render import dataset
+
+
+def _make(args):
+    count, seed, split, degrade = args
+    return dataset(count, seed, split, degrade)
+
+
+def render(count, seed, split="train", degrade=True, workers=4):
+    chunk = 2000
+    jobs = [(min(chunk, count - i), seed * 10_007 + k, split, degrade) for k, i in enumerate(range(0, count, chunk))]
+    with Pool(workers) as pool:
+        parts = pool.map(_make, jobs)
+    return [a for p in parts for a in p[0]], [t for p in parts for t in p[1]]
+
+
+def batches(images, batch_size, rng):
+    """Group lines of similar width (less padding), then shuffle the groups."""
+    order = rng.permutation(len(images))
+    groups = []
+    for s in range(0, len(order), batch_size * 64):
+        block = sorted(order[s:s + batch_size * 64], key=lambda i: images[i].shape[1])
+        groups += [block[k:k + batch_size] for k in range(0, len(block), batch_size)]
+    rng.shuffle(groups)
+    return [g for g in groups if len(g) == batch_size]
+
+
+def evaluate(net, images, texts):
+    net.eval()
+    rec = Recognizer.__new__(Recognizer)
+    rec.net, rec.device = net, "cpu"
+    preds = [t for t, _ in rec.read(images, batch=64)]
+    net.train()
+    return summary(preds, texts)
+
+
+def train(run, rounds=4, per_round=120_000, batch_size=48, lr=2e-3, seed=0, threads=4, resume=False, log=print):
+    run = Path(run)
+    run.mkdir(parents=True, exist_ok=True)
+    torch.manual_seed(seed)
+    torch.set_num_threads(threads)
+    net = LineOCR()
+    steps_per_round = per_round // batch_size
+    opt = torch.optim.AdamW(net.parameters(), lr=lr, weight_decay=1e-4)
+    sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=lr, total_steps=rounds * steps_per_round, pct_start=0.08)
+    start, hist = 0, []
+    ckpt = run / "last.pt"
+    if resume and ckpt.exists():
+        c = torch.load(ckpt, map_location="cpu", weights_only=False)
+        net.load_state_dict(c["state"])
+        opt.load_state_dict(c["opt"])
+        sched.load_state_dict(c["sched"])
+        start, hist = c["round"], c["history"]
+        log(f"resumed after round {start}")
+    val = render(3000, 900_001, "train")       # training fonts and text, unseen seeds
+    test = render(3000, 900_002, "test")       # held-out fonts and held-out text
+    t0 = time.time()
+    for rnd in range(start, rounds):
+        images, texts = render(per_round, seed * 100 + rnd + 1, "train")
+        labels = [torch.tensor(encode(t), dtype=torch.long) for t in texts]
+        rng = np.random.default_rng(seed * 100 + rnd)
+        net.train()
+        running = []
+        for step, idx in enumerate(batches(images, batch_size, rng)[:steps_per_round]):
+            x, lens = batch_tensor([images[i] for i in idx])
+            logp = net(x).transpose(0, 1)  # (T,B,C)
+            tgt = [labels[i] for i in idx]
+            loss = F.ctc_loss(logp, torch.cat(tgt), torch.tensor(lens), torch.tensor([len(t) for t in tgt]),
+                              blank=0, zero_infinity=True)
+            opt.zero_grad()
+            loss.backward()
+            torch.nn.utils.clip_grad_norm_(net.parameters(), 5.0)
+            opt.step()
+            sched.step()
+            running.append(loss.item())
+            if step % 250 == 0:
+                log(f"round {rnd + 1} step {step}/{steps_per_round} loss {np.mean(running[-250:]):.4f} "
+                    f"lr {sched.get_last_lr()[0]:.2e} elapsed {time.time() - t0:.0f}s")
+        entry = {"round": rnd + 1, "train_loss": round(float(np.mean(running[-500:])), 4),
+                 "val_train_fonts": evaluate(net, *val), "test_heldout_fonts": evaluate(net, *test),
+                 "elapsed_s": round(time.time() - t0, 1)}
+        hist.append(entry)
+        log(json.dumps(entry))
+        torch.save({"state": net.state_dict(), "opt": opt.state_dict(), "sched": sched.state_dict(),
+                    "round": rnd + 1, "history": hist}, ckpt)
+        del images, texts, labels
+    info = {"parameters": sum(p.numel() for p in net.parameters()), "rounds": rounds, "lines_per_round": per_round,
+            "batch_size": batch_size, "seed": seed, "history": hist}
+    torch.save({"config": net.config, "state": net.state_dict(), "info": info}, run / "ocr.pt")
+    (run / "train_info.json").write_text(json.dumps(info, indent=2))
+    return net, info

docparse/parsebench.py

@@ -0,0 +1,128 @@
+"""ParseBench (LlamaIndex) integration: registers a `docparse` pipeline with parse-bench.
+
+Usage (parse-bench needs Python 3.12+; install it next to docparse):
+    pip install "parse-bench[fast]"
+    python scripts/parsebench_docparse.py download
+    python scripts/parsebench_docparse.py run docparse            # or: run docparse --test
+
+Each page becomes Markdown (headings, lists, HTML tables) plus layout boxes labelled with
+ParseBench's canonical classes, so all five dimensions can be scored.
+"""
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+from parse_bench.evaluation.layout_adapters.adapters import PyMuPDF4LLMLayoutAdapter
+from parse_bench.extensions import register_layout_adapter, register_pipeline, register_provider
+from parse_bench.inference.providers.base import Provider, ProviderPermanentError
+from parse_bench.schemas.parse_output import LayoutItemIR, LayoutSegmentIR, PageIR, ParseLayoutPageIR, ParseOutput
+from parse_bench.schemas.pipeline import PipelineSpec
+from parse_bench.schemas.pipeline_io import InferenceRequest, InferenceResult, RawInferenceResult
+from parse_bench.schemas.product import ProductType
+
+from .extract import extract, plain, to_markdown
+
+LABELS = {"paragraph": "Text", "list_item": "List-item", "table": "Table", "figure": "Picture",
+          "header": "Page-header", "footer": "Page-footer"}
+
+
+def _label(block):
+    if block["type"] == "heading":
+        return "Title" if block.get("level", 1) == 1 else "Section-header"
+    return LABELS.get(block["type"], "Text")
+
+
+def page_records(doc):
+    """docparse document -> one record per page: body markdown, running header/footer, normalised boxes."""
+    sizes = doc.get("page_sizes") or []
+    pages = []
+    for n in range(1, max(1, doc["pages"]) + 1):
+        blocks = [b for b in doc["blocks"] if b.get("page", 1) == n]
+        boxes, parts, furniture = [], [], {"header": [], "footer": []}
+        number = ""
+        for b in blocks:
+            if b["type"] in furniture:  # page furniture: its own page fields
+                furniture[b["type"]].append(b["text"])
+                if b.get("page_number") and not number:
+                    number = plain(b["text"]).strip()
+                # Also kept in the body inside <page_header>/<page_footer> tags (the ParseBench convention).
+                md = f"<page_{b['type']}>{b['text']}</page_{b['type']}>"
+            else:
+                md = to_markdown({**doc, "blocks": [b], "pages": 1}, tables="html", page_marks=False).strip()
+            start = sum(len(p) + 2 for p in parts)
+            parts.append(md)
+            pos = [start, start + len(md)]
+            if b.get("bbox") and n <= len(sizes):
+                w, h = sizes[n - 1]
+                boxes.append({"class": _label(b), "bbox": list(b["bbox"]), "pos": pos, "text": md,
+                              "width": w, "height": h})
+        w, h = sizes[n - 1] if n <= len(sizes) else (0.0, 0.0)
+        pages.append({"page_index": n - 1, "page_number": n, "text": "\n\n".join(parts), "width": w, "height": h,
+                      "page_boxes": boxes, "header": "\n".join(furniture["header"]),
+                      "footer": "\n".join(furniture["footer"]), "printed_page_number": number})
+    return pages
+
+
+@register_provider("docparse")
+class DocparseProvider(Provider):
+    """Local, CPU-only: exact parsing of the PDF text layer; OCR for images and scanned pages."""
+
+    def run_inference(self, pipeline: PipelineSpec, request: InferenceRequest) -> RawInferenceResult:
+        if request.product_type != ProductType.PARSE:
+            raise ProviderPermanentError(f"docparse only supports PARSE, got {request.product_type}")
+        path = Path(request.source_file_path)
+        if not path.exists():
+            raise ProviderPermanentError(f"File not found: {path}")
+        started = datetime.now()
+        try:
+            doc = extract(path, ocr=self.base_config.get("ocr", "auto"))
+        except Exception as e:  # report and keep the run going
+            raise ProviderPermanentError(f"docparse error: {type(e).__name__}: {e}") from e
+        done = datetime.now()
+        raw = {"pages": page_records(doc), "num_pages": doc["pages"], "warnings": doc["warnings"]}
+        return RawInferenceResult(request=request, pipeline=pipeline, pipeline_name=pipeline.pipeline_name,
+                                  product_type=request.product_type, raw_output=raw, started_at=started,
+                                  completed_at=done, latency_in_ms=int((done - started).total_seconds() * 1000))
+
+    def normalize(self, raw_result: RawInferenceResult) -> InferenceResult:
+        pages, layout = [], []
+        for p in raw_result.raw_output.get("pages", []):
+            md = p["text"]
+            pages.append(PageIR(page_index=p["page_index"], markdown=md))
+            if not (p["width"] and p["height"]):
+                continue
+            items = []
+            for box in p["page_boxes"]:
+                x0, y0, x1, y1 = box["bbox"]
+                W, H = box["width"], box["height"]
+                x0, x1 = max(0.0, min(x0, W)), max(0.0, min(x1, W))
+                y0, y1 = max(0.0, min(y0, H)), max(0.0, min(y1, H))
+                if x1 <= x0 or y1 <= y0:
+                    continue
+                a, b = box["pos"] or (None, None)
+                content = md[a:b] if a is not None else box["text"]
+                seg = LayoutSegmentIR(x=x0 / W, y=y0 / H, w=(x1 - x0) / W, h=(y1 - y0) / H, label=box["class"],
+                                      start_index=a, end_index=b)
+                kind = {"Table": "table", "Picture": "image"}.get(box["class"], "text")
+                items.append(LayoutItemIR(type=kind, md=content, html=content if kind == "table" else "",
+                                          value=content, bbox=seg, layout_segments=[seg]))
+            layout.append(ParseLayoutPageIR(page_number=p["page_number"], width=p["width"], height=p["height"],
+                                            md=md, text=md, items=items, page_header_markdown=p.get("header", ""),
+                                            page_footer_markdown=p.get("footer", ""),
+                                            printed_page_number=p.get("printed_page_number", "")))
+        output = ParseOutput(task_type="parse", example_id=raw_result.request.example_id,
+                             pipeline_name=raw_result.pipeline_name, pages=pages, layout_pages=layout,
+                             markdown="\n\n".join(p.markdown for p in pages))
+        return InferenceResult(request=raw_result.request, pipeline_name=raw_result.pipeline_name,
+                               product_type=raw_result.product_type, raw_output=raw_result.raw_output, output=output,
+                               started_at=raw_result.started_at, completed_at=raw_result.completed_at,
+                               latency_in_ms=raw_result.latency_in_ms)
+
+
+@register_layout_adapter("docparse", priority=90)
+class DocparseLayoutAdapter(PyMuPDF4LLMLayoutAdapter):
+    """Same page-box layout as PyMuPDF4LLM; docparse already emits canonical labels."""
+
+
+register_pipeline(PipelineSpec(pipeline_name="docparse", provider_name="docparse",
+                               product_type=ProductType.PARSE, config={"ocr": "auto"}))

docparse/synth_pages.py

@@ -0,0 +1,148 @@
+"""Synthetic document pages with known content, for end-to-end OCR benchmarks.
+
+A page has a title, section headings, wrapped paragraphs, a bullet list, a ruled table,
+a borderless key/value table and sometimes a two-column section. "scan" pages are
+rotated, blurred, noised and JPEG-compressed after drawing.
+"""
+import io
+import random
+
+import numpy as np
+from PIL import Image, ImageDraw, ImageFilter
+
+from .ocr.render import TEST_FONTS, TRAIN_FONTS, _font, _number, _prose, available, sample_text
+
+PAGE = (1240, 1754)  # A4 at 150 dpi
+
+
+def _family(fonts, r):
+    reg = [f for f in fonts if not any(k in f for k in ("Bold", "Italic", "Oblique", "Light", "Medium", "Semi"))]
+    base = r.choice(reg)
+    bold = next((f for f in fonts if f != base and f.startswith(base.split("-")[0].rsplit(".", 1)[0]) and "Bold" in f
+                 and "Italic" not in f), base)
+    return base, bold
+
+
+def _wrap(text, font, width):
+    lines, cur = [], ""
+    for w in text.split():
+        t = (cur + " " + w).strip()
+        if font.getlength(t) <= width or not cur:
+            cur = t
+        else:
+            lines.append(cur)
+            cur = w
+    return lines + ([cur] if cur else [])
+
+
+def make_page(seed, split="test", scan=False):
+    r = random.Random(seed)
+    fonts = available(TEST_FONTS if split == "test" else TRAIN_FONTS)
+    regular, bold = _family(fonts, r)
+    body = r.randint(20, 26)
+    f_body, f_h1, f_h2 = _font(regular, body), _font(bold, int(body * 1.9)), _font(bold, int(body * 1.4))
+    img = Image.new("L", PAGE, 255)
+    d = ImageDraw.Draw(img)
+    M = 100
+    y = M
+    W = PAGE[0] - 2 * M
+    truth = []
+
+    def text_block(kind, text, font, x, width, level=None):
+        nonlocal y
+        lines = _wrap(text, font, width)
+        lh = int(font.size * 1.35)
+        for ln in lines:
+            d.text((x, y), ln, font=font, fill=0)
+            y += lh
+        b = {"type": kind, "text": " ".join(lines)}
+        if level:
+            b["level"] = level
+        truth.append(b)
+
+    def para(n_lo=120, n_hi=320):
+        return _prose(r, n_lo, n_hi, split)
+
+    text_block("heading", " ".join(w.capitalize() for w in _prose(r, 15, 40, split).split()[:6]), f_h1, M, W, 1)
+    y += body
+    for section in range(r.randint(2, 3)):
+        if y > PAGE[1] - 400:
+            break
+        text_block("heading", f"{section + 1}. " + " ".join(_prose(r, 10, 30, split).split()[:4]).capitalize(), f_h2, M, W, 2)
+        y += body // 2
+        kind = r.choice(["para", "list", "table", "kv", "columns"])
+        if kind == "para" or section == 0:
+            text_block("paragraph", para(), f_body, M, W)
+            y += body
+        if kind == "list":
+            for _ in range(r.randint(2, 4)):
+                text_block("list_item", "• " + _prose(r, 20, 70, split), f_body, M + 20, W - 20)
+            y += body
+        elif kind == "table":
+            cols = r.randint(3, 4)
+            head = ["Item", "Region", "Units", "Amount", "Status", "Date"]
+            r.shuffle(head)
+            rows = [head[:cols]] + [[(sample_text(r, split).split() or ["x"])[0][:12] if c == 0 else _number(r)
+                                     for c in range(cols)] for _ in range(r.randint(3, 5))]
+            cw = [max(f_body.getlength(str(row[c])) for row in rows) + 30 for c in range(cols)]
+            rh = int(body * 1.9)
+            x0 = M
+            if y + rh * len(rows) > PAGE[1] - M:
+                continue
+            for i, row in enumerate(rows):
+                xx = x0
+                for c, v in enumerate(row):
+                    d.text((xx + 15, y + (rh - body) // 2), str(v), font=_font(bold if i == 0 else regular, body), fill=0)
+                    xx += cw[c]
+                d.line([x0, y, xx, y], fill=0, width=2)
+                y += rh
+            d.line([x0, y, x0 + sum(cw), y], fill=0, width=2)
+            xx = x0
+            for c in range(cols + 1):
+                d.line([xx, y - rh * len(rows), xx, y], fill=0, width=2)
+                xx += cw[c] if c < cols else 0
+            truth.append({"type": "table", "rows": [[str(v) for v in row] for row in rows], "ruled": True})
+            y += body * 2
+        elif kind == "kv":
+            keys = r.sample(["Invoice No.", "Date", "Customer", "Total", "Due date", "Reference", "Tax"], 3)
+            rows = [[k, sample_text(r, split)[:24]] for k in keys]
+            lh = int(body * 1.6)
+            for k, v in rows:
+                d.text((M, y), k, font=_font(bold, body), fill=0)
+                d.text((M + 320, y), v, font=f_body, fill=0)
+                y += lh
+            truth.append({"type": "table", "rows": rows, "ruled": False})
+            y += body
+        elif kind == "columns":
+            gap = 60
+            cw = (W - gap) // 2
+            top = y
+            text_block("paragraph", para(250, 400), f_body, M, cw)
+            left_bottom, y = y, top
+            text_block("paragraph", para(250, 400), f_body, M + cw + gap, cw)
+            y = max(y, left_bottom) + body
+    if scan:
+        img = _scan(img, r)
+    return img, truth
+
+
+def _scan(img, r):
+    img = img.rotate(r.uniform(-2.0, 2.0), resample=Image.BICUBIC, fillcolor=255)
+    img = img.filter(ImageFilter.GaussianBlur(r.uniform(0.4, 1.0)))
+    a = np.asarray(img, np.float32)
+    a = a * r.uniform(0.85, 1.0) + r.uniform(0, 20)
+    a = a + np.random.default_rng(r.randint(0, 2 ** 31)).normal(0, r.uniform(4, 10), a.shape)
+    img = Image.fromarray(np.clip(a, 0, 255).astype(np.uint8))
+    buf = io.BytesIO()
+    img.save(buf, "JPEG", quality=r.randint(45, 80))
+    return Image.open(io.BytesIO(buf.getvalue())).convert("L")
+
+
+def truth_text(truth):
+    out = []
+    for b in truth:
+        if b["type"] == "table":
+            out += [" ".join(row) for row in b["rows"]]
+        else:
+            out.append(b["text"])
+    return " ".join(out)

docparse/ui.html

@@ -0,0 +1,71 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>docparse</title>
+<style>
+  :root { --bg:#f6f5f2; --panel:#fff; --ink:#1d1d1f; --muted:#6b6b70; --line:#dedcd6; --accent:#2457c5; }
+  @media (prefers-color-scheme: dark) { :root { --bg:#151517; --panel:#1f1f22; --ink:#ececee; --muted:#9a9aa2; --line:#33333a; --accent:#7ea2ff; } }
+  * { box-sizing: border-box; }
+  body { margin:0; font:15px/1.5 system-ui, sans-serif; background:var(--bg); color:var(--ink); }
+  main { max-width:1100px; margin:0 auto; padding:24px 16px 48px; }
+  h1 { font-size:22px; margin:0 0 4px; }
+  p.sub { color:var(--muted); margin:0 0 20px; }
+  #drop { border:2px dashed var(--line); border-radius:12px; padding:28px; text-align:center; background:var(--panel); cursor:pointer; }
+  #drop.over { border-color:var(--accent); }
+  .opts { display:flex; gap:16px; flex-wrap:wrap; margin:12px 0; color:var(--muted); font-size:14px; }
+  .tabs { display:flex; gap:4px; margin-top:20px; border-bottom:1px solid var(--line); }
+  .tabs button { background:none; border:0; padding:8px 14px; color:var(--muted); font:inherit; cursor:pointer; border-bottom:2px solid transparent; }
+  .tabs button.on { color:var(--ink); border-color:var(--accent); }
+  pre { background:var(--panel); border:1px solid var(--line); border-top:0; margin:0; padding:16px; white-space:pre-wrap; word-break:break-word; max-height:70vh; overflow:auto; font-size:13px; }
+  #status { margin-top:12px; color:var(--muted); min-height:1.5em; }
+  table { border-collapse:collapse; width:100%; background:var(--panel); font-size:13px; }
+  td, th { border:1px solid var(--line); padding:6px 8px; text-align:left; vertical-align:top; }
+  #blocks { max-height:70vh; overflow:auto; }
+</style>
+</head>
+<body>
+<main>
+  <h1>docparse</h1>
+  <p class="sub">Drop a PDF, image, Word, Excel, CSV or PowerPoint file. Born-digital files are parsed exactly; images and scanned pages are read by the small OCR network.</p>
+  <label id="drop">Drop a file here or click to choose<input id="file" type="file" hidden></label>
+  <div class="opts">
+    <label>OCR <select id="ocr"><option value="auto">auto (scanned pages only)</option><option value="always">always</option><option value="never">never</option></select></label>
+    <label><input id="ocrimg" type="checkbox"> read text inside embedded pictures</label>
+  </div>
+  <div id="status"></div>
+  <div class="tabs"><button data-t="md" class="on">Markdown</button><button data-t="blocks">Blocks</button><button data-t="json">JSON</button></div>
+  <pre id="md"></pre><div id="blocks" hidden></div><pre id="json" hidden></pre>
+</main>
+<script>
+const $ = s => document.querySelector(s);
+const drop = $('#drop'), input = $('#file');
+drop.addEventListener('dragover', e => { e.preventDefault(); drop.classList.add('over'); });
+drop.addEventListener('dragleave', () => drop.classList.remove('over'));
+drop.addEventListener('drop', e => { e.preventDefault(); drop.classList.remove('over'); if (e.dataTransfer.files[0]) send(e.dataTransfer.files[0]); });
+input.addEventListener('change', () => input.files[0] && send(input.files[0]));
+document.querySelectorAll('.tabs button').forEach(b => b.onclick = () => {
+  document.querySelectorAll('.tabs button').forEach(x => x.classList.toggle('on', x === b));
+  ['md', 'blocks', 'json'].forEach(id => $('#' + id).hidden = id !== b.dataset.t);
+});
+function esc(s) { return String(s).replace(/[&<>]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;'}[c])); }
+async function send(file) {
+  $('#status').textContent = `Reading ${file.name} ...`;
+  const t0 = performance.now();
+  const r = await fetch('/api/extract', { method: 'POST', body: file, headers: {
+    'X-Filename': file.name, 'X-OCR': $('#ocr').value, 'X-OCR-Images': $('#ocrimg').checked ? '1' : '0' } });
+  const data = await r.json();
+  if (!r.ok) { $('#status').textContent = data.error; return; }
+  const d = data.document;
+  $('#status').textContent = `${d.blocks.length} blocks, ${d.pages} page(s)/sheet(s) in ${((performance.now() - t0) / 1000).toFixed(1)} s` +
+    (d.warnings.length ? ' · ' + d.warnings.join('; ') : '');
+  $('#md').textContent = data.markdown;
+  $('#json').textContent = JSON.stringify(d, null, 2);
+  const rows = d.blocks.map(b => `<tr><td>${b.page ?? b.sheet ?? ''}</td><td>${b.type}${b.level ? ' ' + b.level : ''}</td><td>${b.source ?? ''}</td>` +
+    `<td>${b.confidence ?? ''}</td><td>${b.type === 'table' ? esc(b.rows.map(r => r.join(' | ')).join('\n')).replace(/\n/g, '<br>') : esc(b.text ?? '')}</td></tr>`);
+  $('#blocks').innerHTML = `<table><tr><th>page</th><th>type</th><th>source</th><th>conf.</th><th>content</th></tr>${rows.join('')}</table>`;
+}
+</script>
+</body>
+</html>

docparse/ui.py

@@ -0,0 +1,56 @@
+"""Local browser UI: upload a document, see Markdown, JSON and the blocks (standard library HTTP server)."""
+import json
+import tempfile
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from threading import Lock
+
+PAGE = Path(__file__).with_name("ui.html")
+MAX_BYTES = 40 * 2 ** 20
+LOCK = Lock()  # one extraction at a time keeps memory and CPU bounded
+
+
+def serve(port=7862, host="127.0.0.1", model=None):
+    from .extract import extract, to_markdown
+
+    class Handler(BaseHTTPRequestHandler):
+        def _send(self, code, body, ctype="application/json"):
+            data = body if isinstance(body, bytes) else body.encode()
+            self.send_response(code)
+            self.send_header("Content-Type", ctype)
+            self.send_header("Content-Length", str(len(data)))
+            self.end_headers()
+            self.wfile.write(data)
+
+        def do_GET(self):
+            if self.path in ("/", "/index.html"):
+                self._send(200, PAGE.read_bytes(), "text/html; charset=utf-8")
+            else:
+                self._send(404, json.dumps({"error": "not found"}))
+
+        def do_POST(self):
+            if not self.path.startswith("/api/extract"):
+                return self._send(404, json.dumps({"error": "not found"}))
+            n = int(self.headers.get("Content-Length") or 0)
+            if n <= 0 or n > MAX_BYTES:
+                return self._send(413, json.dumps({"error": f"file must be 1 byte to {MAX_BYTES // 2 ** 20} MB"}))
+            name = Path(self.headers.get("X-Filename", "upload.bin")).name
+            ocr = self.headers.get("X-OCR", "auto")
+            if ocr not in ("auto", "always", "never"):
+                ocr = "auto"
+            data = self.rfile.read(n)
+            try:
+                with LOCK, tempfile.TemporaryDirectory() as tmp:
+                    path = Path(tmp) / name
+                    path.write_bytes(data)
+                    doc = extract(path, ocr=ocr, ocr_images=self.headers.get("X-OCR-Images") == "1", model=model)
+                self._send(200, json.dumps({"markdown": to_markdown(doc), "document": doc}, default=str))
+            except Exception as exc:  # report parse errors to the page instead of dropping the connection
+                self._send(400, json.dumps({"error": f"{type(exc).__name__}: {exc}"}))
+
+        def log_message(self, *a):
+            pass
+
+    srv = ThreadingHTTPServer((host, port), Handler)
+    print(f"docparse UI on http://{host}:{port}", flush=True)
+    srv.serve_forever()

packages.txt

@@ -0,0 +1,3 @@
+libreoffice-writer
+libreoffice-calc
+libreoffice-impress

requirements.txt

@@ -0,0 +1,7 @@
+torch==2.8.0
+numpy>=1.26,<3
+pillow>=10,<12
+pdfplumber>=0.11
+pypdfium2>=4.30
+python-docx>=1.1
+openpyxl>=3.1