- Parenthetical blocks (Author year; Author year) now expand into individual
cite spans so each is independently clickable rather than linking only to the
first author
- Detected citation patterns always emit a cite segment regardless of whether
the reference is in citeIndex; empty-refs spans fall back to a fielded ADS
search (first_author:"Surname" year:YEAR) on click
- Matched corpus citations get a rubric-tinted dotted underline; unmatched get
a grey dotted underline; both turn rubric on hover
- Cite elements changed from <span> to <button> for accessibility
- Add .vite to frontend .gitignore; fix FakeCorpus test stub (arxiv_ids/bibcodes)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix: preserve non-referee cache key format — only add ctx_hash element for referee mode
- _flush_pending_as_para: snapshot and clear pending_headings before the
length check so short-text flushes don't leave headings to leak into the
next paragraph
- Level-aware orphan pruning: when a new heading arrives with no pending
prose, drop any accumulated pending_headings at the same or deeper level
(sibling/child orphans) while keeping true parent headings so consecutive
sub-headings still chain correctly
- Track pending_heading_levels in parallel to enable level-aware pruning;
clear it in all three sites where pending_headings is cleared
- Strip trailing '*' from list-item heading captures (parity with _HEADING path)
- Add test_heading_not_leaked_from_empty_section to cover the leakage case
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bootstrap.py: ensure_corpus() downloads corpus from HF dataset repo on first
run; no-op when both corpus_dir and faiss_path already exist
- main.py: replace deprecated @app.on_event("startup") with lifespan context
manager; wire in ensure_corpus before get_corpus; mount frontend/dist at /
for SPA serving (API routes registered first, take precedence)
- Dockerfile: two-stage build (node:20-slim → python:3.11-slim + uv); all
runtime data under /data (corpus, faiss, figures, sqlite, HF cache)
- .dockerignore: excludes node_modules, backend/data, .venv, prototype/, docs/
- README.md: prepend HF Spaces YAML frontmatter (docker SDK, port 7860)
- pyproject.toml: add huggingface-hub>=0.23 as explicit dependency
- tests/test_bootstrap.py: 3 new tests (no-op + download paths); 70 total green
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Merge build-v1: Beacon — annotated astronomy paper reader
Full production build from the HTML prototype: FastAPI backend (PDF/text
parsing, Pathfinder corpus retrieval via mmap'd fp16 FAISS, multi-provider
streaming LLM proxy with completion cache, figure extraction) and React/Vite
frontend (typography-faithful reader with streaming margin notes, citation
and figure popovers, sessions, display preferences). 67 backend + 21
frontend tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add litCount (3–10, default 3) to DISPLAY_DEFAULTS and DisplayMenu range slider; slice lit at generate() time (changes annotator cache keys by design) and at MarginNote render time via tweaks prop
- Accept .txt/.md/.markdown files in Upload drop zone; update subline copy and add ap-splash-about block with prose and how-to mono line
- Rename user-visible "astroparse" → "beacon": wordmark (Upload, Pipeline), toolbar mark, <title>, session download suffix (.beacon.json); localStorage keys and CSS ap-* classes intentionally unchanged
- package.json name → beacon-frontend
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Raise retrieve_for_paragraphs default top_k from 3 to 10 so the backend
always returns up to ten ranked works per paragraph/figure (display slice
is controlled by the frontend preference). Add parse_text() for .txt/.md/.markdown
uploads: UTF-8 decode → segment_markdown, title from first heading or filename
stem, pages=0, figures=[], all seven pipeline stages still emitted. Tests: +1
top-k assertion on a 12-row FakeDataset, +8 parse_text unit tests, +1 API
integration test for markdown upload (66 total, all green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: scrollable popovers; default annotation mode is historical context
- Bug A: pass force_text=False to pymupdf4llm.to_markdown (stops picture-text
emission at source) plus stateful strip_picture_text() filter as belt-and-
suspenders; handles both 'Start/End' (1.27.x) and 'Begin/End' marker variants.
Also add _RUNNING_HEADER filter in strip_page_artifacts to catch all-caps
page-header lines that survive remove_repeating_lines on single-page variants.
- Bug B: expose demarkdown() function that strips **bold**, _emphasis_, backticks,
repairs digit-dot and digit-x spacing artifacts, and converts bracketed exponents
(10[5] -> 10^5) to unicode superscripts. Applied to paragraph text and figure
captions (figures.py).
- Bug B2: add _prev_para_is_open() / _OPEN_FUNCTION_WORDS heuristic: merge a
block into the previous paragraph when the previous text ends with a syntactically
open function word (and, or, in, of, the, ...). Digit-ended and hyphen-ended
blocks are excluded to avoid false positives on 2-column line-number artifacts.
- 8 new T…
fix: bypass pymupdf4llm layout analysis on vector-heavy pages
Page 3 of s41586-024-08293-0.pdf contains a 65 MB content stream (563k vector
paths — a JWST image drawn as vectors). pymupdf4llm's layout/table analysis
takes 400+s on it; plain page.get_text("blocks") takes 0.8s and recovers all
text.
- Add _MAX_PAGE_STREAM_BYTES = 8_000_000 constant and _pathological_pages()
helper in parse.py: detects oversized content streams via page.read_contents()
- In parse_pdf: call pymupdf4llm.to_markdown with pages=[normal only], then
interleave fallback chunks (get_text("blocks") joined with \\n\\n) for bad pages;
assert len(chunks)==n_pages after splicing
- In figures.py extract_figures: skip cluster_drawings() for pathological pages
(raster image rects still used); import _MAX_PAGE_STREAM_BYTES from parse
- Add 4 tests: 3 unit tests for _pathological_pages (duck-typed fake docs) and
1 integration test on s41586 asserting < 60s and >= 30 paragraphs
Result: s41586 parses in 4.5s (was unparseable), 71 paragraphs; all 7 test
papers under 10s; ext…
feat: figure mention popovers with caption annotator
Implements Task 20: clickable Fig./Figure N mentions in paragraph text open a
FigurePopover with the figure image, caption, and five-mode streaming annotator,
reusing CitePopover chrome and NoteBody extracted from MarginNote. Adds the 7th
pipeline stage row (vii · Tipping in the plates · pymupdf), extends session
round-trip with figures/litByFig, and generalises generate() to resolve both
paragraph and figure targets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test: make figures traversal test exercise actual traversal payloads
Adds backend figure extraction using pymupdf (extract_figures in figures.py),
a new Figure schema, a 7th pipeline stage ("Tipping in the plates"), batched
embedding of paragraph + caption texts in a single OpenAI call, litByFig
retrieval, and a validated GET /api/figures/{pdfHash}/{figId}.png endpoint.
Test suite grows from 31 to 44 (all green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs: figures & captions spec addendum and plan (tasks 19-21)
- Add _AFFILIATION regex to strip numbered blockquote affiliation lines (> 1 _..._)
from blocks during segment_markdown, preventing them from merging into paragraphs
- Add _is_author_affiliation_block detector (≥3 commas AND prose fraction < 0.25)
applied only to first 3 front-matter candidates, to drop author name blocks
- Add _clean_authors helper that strips [..], (..), _* chars, keeps tokens with ≥2
letters, and truncates to first 3 authors + ' et al.' when more than 3 remain
- Apply _clean_authors in parse_pdf to the extracted raw author line
- Add 3 new tests (test_segment_drops_author_affiliation_block,
test_clean_authors_three_plus_et_al, test_clean_authors_short_list_unchanged);
suite: 31 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: parse CRLF-framed SSE events from sse-starlette
The browser pipeline screen stalled at the first stage: parseSseChunk
split events on '\n\n' but sse-starlette frames with '\r\n\r\n', so
no event ever parsed. Accept CRLF/LF separators per the SSE spec.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full HF dataset (20 GB, float64 embeds, citation graphs) exceeds this
machine's disk and RAM. build_corpus.py converts the downloaded shards to
text-only parquet (~1.3 GB) plus a float16 IndexScalarQuantizer (~3.8 GB),
deleting source shards as consumed. Corpus.load() now mmaps the index.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires toolbar save/load, localStorage autosave, and a DisplayMenu
popover for palette, body size, justify, and notes-side preferences.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: move generate calls out of setState updaters
Use a commentaryRef (assigned each render) so onSetMode and onSetAllModes
can check commentary presence without calling generate() inside a setState
updater — eliminating the React StrictMode double-invocation anti-pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat: streaming annotation generation and annotator panel
Implements Task 16: wires annotateStream SSE into App.jsx generate()
with inflight deduplication, adds AnnotatorPanel (three provider chips,
localStorage persist), Toast component (2.6s auto-dismiss), streaming
display in MarginNote (text renders as first token arrives), and a
trailing 150ms throttle on Reader's recompute during token streaming.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports UploadScreen and PipelineScreen from prototype/pipeline.jsx as
React components; replaces App.jsx placeholder with a state machine
(upload → pipeline → reader) connected to startParse/jobEvents/importSession.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire parse pipeline as background job with SSE progress (six-stage events)
and annotate proxy with SQLite caching + token streaming; fix Anthropic
mid-stream error events and malformed JSON skip in all provider branches;
add max_tokens=400 to OpenAI branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real corpus schema has a dedicated arxiv_id column; ads_id is a numeric
ADS-internal identifier. Also suppress old-style pre-2007 arXiv IDs
(astro-ph9804... format) via the existing YYYY.NNNN regex guard.
Update FakeDataset rows in test to mirror real corpus field names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds Corpus class (FAISS-backed HuggingFace dataset wrapper), embed_batch,
_lit_from_row, and retrieve_for_paragraphs — the core literature retrieval
pipeline that ranks candidate papers per paragraph using embed+keyword scores.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat: keyword extraction and pathfinder keyword weighting
Add segment_markdown and parse_pdf to turn a PDF into a Paper with clean
prose paragraphs: strips figure captions, picture placeholders, and
markdown tables; merges page-break continuation fragments; stops at
References; fixes strip_page_artifacts to preserve blank lines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
