- _flush_pending_as_para: snapshot and clear pending_headings before the
length check so short-text flushes don't leave headings to leak into the
next paragraph
- Level-aware orphan pruning: when a new heading arrives with no pending
prose, drop any accumulated pending_headings at the same or deeper level
(sibling/child orphans) while keeping true parent headings so consecutive
sub-headings still chain correctly
- Track pending_heading_levels in parallel to enable level-aware pruning;
clear it in all three sites where pending_headings is cleared
- Strip trailing '*' from list-item heading captures (parity with _HEADING path)
- Add test_heading_not_leaked_from_empty_section to cover the leakage case
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Raise retrieve_for_paragraphs default top_k from 3 to 10 so the backend
always returns up to ten ranked works per paragraph/figure (display slice
is controlled by the frontend preference). Add parse_text() for .txt/.md/.markdown
uploads: UTF-8 decode → segment_markdown, title from first heading or filename
stem, pages=0, figures=[], all seven pipeline stages still emitted. Tests: +1
top-k assertion on a 12-row FakeDataset, +8 parse_text unit tests, +1 API
integration test for markdown upload (66 total, all green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bug A: pass force_text=False to pymupdf4llm.to_markdown (stops picture-text
emission at source) plus stateful strip_picture_text() filter as belt-and-
suspenders; handles both 'Start/End' (1.27.x) and 'Begin/End' marker variants.
Also add _RUNNING_HEADER filter in strip_page_artifacts to catch all-caps
page-header lines that survive remove_repeating_lines on single-page variants.
- Bug B: expose demarkdown() function that strips **bold**, _emphasis_, backticks,
repairs digit-dot and digit-x spacing artifacts, and converts bracketed exponents
(10[5] -> 10^5) to unicode superscripts. Applied to paragraph text and figure
captions (figures.py).
- Bug B2: add _prev_para_is_open() / _OPEN_FUNCTION_WORDS heuristic: merge a
block into the previous paragraph when the previous text ends with a syntactically
open function word (and, or, in, of, the, ...). Digit-ended and hyphen-ended
blocks are excluded to avoid false positives on 2-column line-number artifacts.
- 8 new T…
fix: bypass pymupdf4llm layout analysis on vector-heavy pages
Page 3 of s41586-024-08293-0.pdf contains a 65 MB content stream (563k vector
paths — a JWST image drawn as vectors). pymupdf4llm's layout/table analysis
takes 400+s on it; plain page.get_text("blocks") takes 0.8s and recovers all
text.
- Add _MAX_PAGE_STREAM_BYTES = 8_000_000 constant and _pathological_pages()
helper in parse.py: detects oversized content streams via page.read_contents()
- In parse_pdf: call pymupdf4llm.to_markdown with pages=[normal only], then
interleave fallback chunks (get_text("blocks") joined with \\n\\n) for bad pages;
assert len(chunks)==n_pages after splicing
- In figures.py extract_figures: skip cluster_drawings() for pathological pages
(raster image rects still used); import _MAX_PAGE_STREAM_BYTES from parse
- Add 4 tests: 3 unit tests for _pathological_pages (duck-typed fake docs) and
1 integration test on s41586 asserting < 60s and >= 30 paragraphs
Result: s41586 parses in 4.5s (was unparseable), 71 paragraphs; all 7 test
papers under 10s; ext…
fix: drop author/affiliation front matter; show first three authors et al.
- Add _AFFILIATION regex to strip numbered blockquote affiliation lines (> 1 _..._)
from blocks during segment_markdown, preventing them from merging into paragraphs
- Add _is_author_affiliation_block detector (≥3 commas AND prose fraction < 0.25)
applied only to first 3 front-matter candidates, to drop author name blocks
- Add _clean_authors helper that strips [..], (..), _* chars, keeps tokens with ≥2
letters, and truncates to first 3 authors + ' et al.' when more than 3 remain
- Apply _clean_authors in parse_pdf to the extracted raw author line
- Add 3 new tests (test_segment_drops_author_affiliation_block,
test_clean_authors_three_plus_et_al, test_clean_authors_short_list_unchanged);
suite: 31 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add segment_markdown and parse_pdf to turn a PDF into a Paper with clean
prose paragraphs: strips figure captions, picture placeholders, and
markdown tables; merges page-break continuation fragments; stops at
References; fixes strip_page_artifacts to preserve blank lines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
