Classify a blocked Scholar page before waiting, not after

#8 made the classification correct but left it arriving 30s late. Measured
against the deployment: the error now says "Google Scholar blocked the
request", but it still takes 30.7s, because every signal was consulted
only in the selector-timeout handler.

All of them are available the moment navigation completes, so they are
checked there instead. A page carrying no results at all is decidable
immediately: Scholar is server-rendered, so a genuine results page already
has div.gs_ri in the DOM.

That "no results" guard is also what makes the wording check safe this
early - and it caught a real false positive during this change. Without
it, a legitimate search for "unusual traffic" returns papers whose
snippets quote the interstitial and was misreported as a block. The test
written for that risk failed against the first version of this commit,
which is the whole reason it exists.

The block exception now carries the evidence that classified it -
redirect URL, challenge element, or wording. Diagnostics had only been
added to the unclassified path in #8, which is precisely the path that did
not fire in production, so "blocked" was an assertion with nothing behind
it and no way to tell which signal had matched.

Mutation testing also showed the explicit up-front URL check was dead
code: removing it changed no test, because the evidence check below
already consults the URL and a /sorry page never has results. Deleted
rather than kept as an untested branch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSNSYnceqvdVz7Csis4K9e

CONTRIBUTING.md

@@ -30,7 +30,7 @@ ruff check .   # same lint CI runs
 pytest
 ```
 
-260 tests, a few seconds, fully offline. All outbound HTTP is mocked with
+264 tests, a few seconds, fully offline. All outbound HTTP is mocked with
 `respx`; the Playwright-driven scrapers (Bing, Brave, Google Scholar,
 Google Patents search) run against a real headless Chromium but navigate to
 local fixture HTML instead of the live sites — see `tests/helpers.py` for

serp.py

@@ -58,17 +58,24 @@ class BraveSearchBlockedException(Exception):
 class GoogleScholarBlockedException(Exception):
     """Raised when Google Scholar serves its anti-bot interstitial.
 
-    Scholar refuses datacenter IPs with a /sorry challenge page rather than
-    results. That page simply has no `div.gs_ri`, so without this the
-    scraper waits out the full selector timeout and then reports a bare
-    "Timeout exceeded" - slow, and indistinguishable from a genuine
-    selector regression.
+    Scholar refuses datacenter IPs with a challenge page rather than
+    results. That page has no `div.gs_ri`, so without this the scraper
+    waits out the full selector timeout and then reports a bare "Timeout
+    exceeded" - slow, and indistinguishable from a genuine selector
+    regression.
+
+    Carries the evidence that classified it. Without that, "blocked" is an
+    assertion the reader has no way to check, and there is no way to tell
+    which signal fired - which is exactly what made the first two attempts
+    at this hard to debug from the deployment.
     """
 
-    def __init__(self, *args):
-        super().__init__(
-            "Google Scholar blocked the request (anti-bot interstitial served "
-            "instead of results); the deployment's IP is likely rate-limited.")
+    def __init__(self, evidence: str = ""):
+        message = ("Google Scholar blocked the request (anti-bot interstitial served "
+                   "instead of results); the deployment's IP is likely rate-limited.")
+        if evidence:
+            message = f"{message} Evidence: {evidence}"
+        super().__init__(message)
 
 
 class GoogleScholarUnavailableError(Exception):
@@ -217,29 +224,54 @@ async def _extract_google_scholar_results(page: Page, n_results: int,
     """
     url = final_url if final_url is not None else getattr(page, "url", None)
 
-    async def _blocked() -> bool:
+    async def _block_evidence() -> Optional[str]:
+        """Which signal, if any, says this is a challenge page."""
         if _url_looks_blocked(url):
-            return True
+            return f"redirected to {url!r}"
         try:
             if await page.locator(_SCHOLAR_BLOCK_SELECTOR).count():
-                return True
+                return "challenge element present in the page"
         except Exception:
             pass
-        return _looks_like_scholar_block(await page.content())
+        if _looks_like_scholar_block(await page.content()):
+            return "page wording matches Google's interstitial"
+        return None
+
+    # Classify before waiting on anything. A page carrying no results at
+    # all is decidable the moment navigation completes - Scholar is
+    # server-rendered, so a genuine results page already has div.gs_ri in
+    # the DOM - and every signal (redirect URL, challenge element, Google's
+    # wording) is available now. Doing this here rather than in the timeout
+    # handler is what stops a block costing a full selector wait: measured
+    # against the deployment, classification was already correct but
+    # arrived 30s late for exactly this reason.
+    #
+    # The "no results" guard is also what makes the wording check safe this
+    # early: a real search for "unusual traffic" returns papers, so it
+    # never reaches here.
+    if await page.locator(_SCHOLAR_RESULT_SELECTOR).count() == 0:
+        early_evidence = await _block_evidence()
+        if early_evidence:
+            raise GoogleScholarBlockedException(early_evidence)
 
     try:
         await page.wait_for_selector(
             f"{_SCHOLAR_RESULT_SELECTOR}, {_SCHOLAR_BLOCK_SELECTOR}",
             state="attached", timeout=timeout_ms)
     except PlaywrightTimeoutError:
-        if await _blocked():
-            raise GoogleScholarBlockedException() from None
+        evidence = await _block_evidence()
+        if evidence:
+            raise GoogleScholarBlockedException(evidence) from None
         raise GoogleScholarUnavailableError(
             "Google Scholar returned neither results nor a recognisable "
             f"challenge page ({await _describe_page(page, url)})") from None
 
-    if await page.locator(_SCHOLAR_RESULT_SELECTOR).count() == 0 or await _blocked():
-        raise GoogleScholarBlockedException()
+    # Results win. Only a page with none of them gets classified, which
+    # keeps the wording check from firing on a legitimate search whose
+    # snippets happen to quote Google's interstitial.
+    if await page.locator(_SCHOLAR_RESULT_SELECTOR).count() == 0:
+        raise GoogleScholarBlockedException(
+            await _block_evidence() or "no results on the page")
 
     items = await page.locator(_SCHOLAR_RESULT_SELECTOR).all()
     results = []

tests/test_serp_playwright.py

@@ -283,11 +283,51 @@ async def test_a_sorry_redirect_url_counts_as_a_block(page_factory):
     """Google redirects blocked clients to /sorry/index. The final URL is
     markup-independent evidence, unlike any selector we can guess at."""
     page = await page_factory("<html><body><p>nothing recognisable here</p></body></html>")
-    page.url_override = "https://scholar.google.com/sorry/index?continue=..."
 
     with pytest.raises(GoogleScholarBlockedException):
         await _extract_google_scholar_results(
-            page, 10, timeout_ms=1000, final_url=page.url_override)
+            page, 10, timeout_ms=1000,
+            final_url="https://scholar.google.com/sorry/index?continue=...")
+
+
+async def test_a_sorry_redirect_is_recognised_without_any_waiting(page_factory):
+    """The URL is known before the page is even inspected, so a /sorry
+    redirect must not cost a selector wait.
+
+    Measured against the live deployment, this was the whole remaining
+    problem: classification was correct but arrived only after the full
+    30s timeout, because the URL was checked in the timeout handler rather
+    than up front. A long timeout is passed deliberately - the assertion
+    is that it is never reached.
+    """
+    import time
+
+    page = await page_factory("<html><body><p>nothing recognisable here</p></body></html>")
+
+    started = time.perf_counter()
+    with pytest.raises(GoogleScholarBlockedException):
+        await _extract_google_scholar_results(
+            page, 10, timeout_ms=30_000,
+            final_url="https://scholar.google.com/sorry/index?continue=...")
+    elapsed = time.perf_counter() - started
+
+    assert elapsed < 2, f"waited {elapsed:.1f}s on a URL that was known up front"
+
+
+async def test_a_block_reports_the_page_it_saw(page_factory):
+    """Diagnostics were added only to the unclassified path, which is the
+    path that did not fire in production. The block path needs them too -
+    otherwise "blocked" is an assertion the reader cannot check, and there
+    is no way to tell which signal fired.
+    """
+    page = await page_factory("<html><body><p>nothing recognisable here</p></body></html>")
+
+    with pytest.raises(GoogleScholarBlockedException) as exc_info:
+        await _extract_google_scholar_results(
+            page, 10, timeout_ms=1000,
+            final_url="https://scholar.google.com/sorry/index?continue=xyz")
+
+    assert "sorry" in str(exc_info.value), "the evidence must appear in the message"
 
 
 async def test_normal_scholar_results_are_unaffected(page_factory):
@@ -313,3 +353,42 @@ async def test_bing_result_hrefs_are_decoded(page_factory):
         "https://docs.python.org/3/library/asyncio.html",
         "https://example.org/direct",
     ]
+
+
+async def test_a_challenge_page_with_only_wording_is_also_recognised_immediately(page_factory):
+    """Don't leave the fast path depending on which signal happens to fire.
+
+    A page with no results at all, whose wording matches Google's
+    interstitial, is decidable the moment it loads - Scholar is
+    server-rendered, so a genuine results page has div.gs_ri in the DOM as
+    soon as navigation completes. Requiring "no results present" is what
+    makes the wording check safe this early: a real search for "unusual
+    traffic" returns papers, so it never reaches this branch.
+    """
+    import time
+
+    page = await page_factory(
+        "<html><body><p>Our systems have detected unusual traffic from your "
+        "computer network.</p></body></html>")
+
+    started = time.perf_counter()
+    with pytest.raises(GoogleScholarBlockedException):
+        await _extract_google_scholar_results(page, 10, timeout_ms=30_000)
+    elapsed = time.perf_counter() - started
+
+    assert elapsed < 2, f"waited {elapsed:.1f}s on a page that was decidable up front"
+
+
+async def test_a_real_results_page_mentioning_unusual_traffic_is_not_a_block(page_factory):
+    """The guard on the branch above, stated as a test: results win over
+    wording, so a legitimate search whose snippets contain the interstitial
+    phrasing is never misreported as a block."""
+    page = await page_factory(
+        '<html><body><div class="gs_ri"><h3><a href="https://example.org/p">'
+        "Detecting unusual traffic in networks</a></h3>"
+        '<div class="gs_rs">Our systems have detected unusual traffic patterns.</div>'
+        "</div></body></html>")
+
+    results = await _extract_google_scholar_results(page, 10)
+
+    assert [r["title"] for r in results] == ["Detecting unusual traffic in networks"]