Spaces:
Sleeping
Sleeping
Commit History
-
Detect Google Scholar's block page, and unwrap Bing's redirect links
Claude +1: Claude Opus 5Unverified signatureBoth found by running the live smoke test against the deployed MCP server, not by the suite - the division of labour CONTRIBUTING.md already describes. Google Scholar serves an anti-bot interstitial to the Space's IP instead of results. That page has no `div.gs_ri`, so the scraper sat in `wait_for_selector` for the full 30 seconds and then reported a bare "Timeout 30000ms exceeded" - slow for every caller, and worded like a broken selector rather than a blocked deployment. It now waits for the results selector and Google's challenge elements together, so a block is recognised as soon as the page loads and raises GoogleScholarBlockedException with a message that says what actually happened. A challenge page carrying none of those elements falls back to matching Google's wording, but only after the selector wait has failed: sniffing the text up front would be faster and wrong, since a legitimate Scholar search for "unusual traffic" returns papers whose snippets contain that phrase. A p…f493ed5
-
Add pytest harness with characterization tests for scrap/ops/serp/utils
ClaudeUnverified signatureAdds a test safety net ahead of any refactoring, per the TDD/architecture review of this codebase (no test infrastructure existed before this): - pytest + pytest-asyncio + respx as dev-only deps (requirements-dev.txt) - Pure-function tests for SerpQuery's n_results clamping, utils.py's exception logging, and every OPS JSON-walking helper (ops.py handles OPS's XML-to-JSON quirk where a repeated element is a list but a single one stays a bare dict - tests cover both shapes for each helper) - HTTP-mocked (respx) tests for scrap_patent_async's HTML parsing (abstract/description/claims/classification-code extraction, plus the documented ValueError path for an unexpected page layout), the OPS token manager's caching/refresh/retry-on-stale-token behavior, and query_arxiv's Atom feed parsing - A DDGS-mocked test pinning the DuckDuckGo empty-results-means-blocked fix from d341a7a, so a regression there fails a test instead of shipping silently - Real-headless-Chromium tests f…5796881