docs(plan): Task 5 must preserve blank lines in strip_page_artifacts
Browse files
docs/superpowers/plans/2026-06-10-astroparse.md
CHANGED
|
@@ -346,6 +346,29 @@ curl -L -o backend/tests/fixtures/mowla_iyer_2024.pdf https://arxiv.org/pdf/2402
|
|
| 346 |
```
|
| 347 |
Verify it's a PDF: `file backend/tests/fixtures/mowla_iyer_2024.pdf` → "PDF document".
|
| 348 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 349 |
- [ ] **Step 2: Write failing tests**
|
| 350 |
|
| 351 |
```python
|
|
|
|
| 346 |
```
|
| 347 |
Verify it's a PDF: `file backend/tests/fixtures/mowla_iyer_2024.pdf` → "PDF document".
|
| 348 |
|
| 349 |
+
- [ ] **Step 1b: Fix `strip_page_artifacts` to preserve blank lines** (integration fix found in Task 4 review: `parse_pdf` rejoins lines and segments on blank lines, so artifact-stripping must NOT drop them). TDD: add to `backend/tests/test_parse_clean.py`:
|
| 350 |
+
|
| 351 |
+
```python
|
| 352 |
+
def test_strip_page_artifacts_preserves_blank_lines():
|
| 353 |
+
lines = ["First para.", "", "12", "Second para."]
|
| 354 |
+
assert strip_page_artifacts(lines) == ["First para.", "", "Second para."]
|
| 355 |
+
```
|
| 356 |
+
|
| 357 |
+
Run (FAIL), then change the filter in `strip_page_artifacts` so blank lines pass through (drop only artifacts):
|
| 358 |
+
|
| 359 |
+
```python
|
| 360 |
+
def strip_page_artifacts(lines: list[str]) -> list[str]:
|
| 361 |
+
return [
|
| 362 |
+
l for l in lines
|
| 363 |
+
if not l.strip()
|
| 364 |
+
or (not _PAGE_NUM.match(l.strip())
|
| 365 |
+
and not _ARXIV_STAMP.search(l)
|
| 366 |
+
and not _DOI_STAMP.match(l.strip()))
|
| 367 |
+
]
|
| 368 |
+
```
|
| 369 |
+
|
| 370 |
+
Run (PASS — the existing 3 tests must also still pass; their inputs contain no blank lines).
|
| 371 |
+
|
| 372 |
- [ ] **Step 2: Write failing tests**
|
| 373 |
|
| 374 |
```python
|