Classify a blocked Scholar page before waiting, not after (#9)

#8 made the classification correct but left it arriving 30s late. Measured
against the deployment: the error said "Google Scholar blocked the
request", but still took 30.7s, because every signal was consulted only in
the selector-timeout handler.

All of them are available the moment navigation completes, so they are
checked there instead - and deliberately all of them, not just the one
suspected of firing in production, since a fix contingent on that guess
would repeat the mistake from #7. A page carrying no results at all is
decidable immediately: Scholar is server-rendered, so a genuine results
page already has div.gs_ri in the DOM.

That "no results" guard is what makes the wording check safe this early,
and it caught a real false positive during this change: without it, a
legitimate search for "unusual traffic" returns papers whose snippets
quote the interstitial and was misreported as a block.

The block exception now carries the evidence that classified it.
Diagnostics had only been added to the unclassified path in #8 - precisely
the path that did not fire in production.

Mutation testing also showed the explicit up-front URL check was dead
code, since the evidence check below already consults the URL and a /sorry
page never has results. Deleted rather than kept as an untested branch.

264 tests, lint clean, timing asserted rather than assumed.

CONTRIBUTING.md

@@ -30,7 +30,7 @@ ruff check .   # same lint CI runs
 pytest
 ```
 
-260 tests, a few seconds, fully offline. All outbound HTTP is mocked with
+264 tests, a few seconds, fully offline. All outbound HTTP is mocked with
 `respx`; the Playwright-driven scrapers (Bing, Brave, Google Scholar,
 Google Patents search) run against a real headless Chromium but navigate to
 local fixture HTML instead of the live sites — see `tests/helpers.py` for

serp.py

@@ -58,17 +58,24 @@ class BraveSearchBlockedException(Exception):
 class GoogleScholarBlockedException(Exception):
     """Raised when Google Scholar serves its anti-bot interstitial.
 
-    Scholar refuses datacenter IPs with a /sorry challenge page rather than
-    results. That page simply has no `div.gs_ri`, so without this the
-    scraper waits out the full selector timeout and then reports a bare
-    "Timeout exceeded" - slow, and indistinguishable from a genuine
-    selector regression.
+    Scholar refuses datacenter IPs with a challenge page rather than
+    results. That page has no `div.gs_ri`, so without this the scraper
+    waits out the full selector timeout and then reports a bare "Timeout
+    exceeded" - slow, and indistinguishable from a genuine selector
+    regression.
+
+    Carries the evidence that classified it. Without that, "blocked" is an
+    assertion the reader has no way to check, and there is no way to tell
+    which signal fired - which is exactly what made the first two attempts
+    at this hard to debug from the deployment.
     """
 
-    def __init__(self, *args):
-        super().__init__(
-            "Google Scholar blocked the request (anti-bot interstitial served "
-            "instead of results); the deployment's IP is likely rate-limited.")
+    def __init__(self, evidence: str = ""):
+        message = ("Google Scholar blocked the request (anti-bot interstitial served "
+                   "instead of results); the deployment's IP is likely rate-limited.")
+        if evidence:
+            message = f"{message} Evidence: {evidence}"
+        super().__init__(message)
 
 
 class GoogleScholarUnavailableError(Exception):
@@ -217,29 +224,54 @@ async def _extract_google_scholar_results(page: Page, n_results: int,
     """
     url = final_url if final_url is not None else getattr(page, "url", None)
 
-    async def _blocked() -> bool:
+    async def _block_evidence() -> Optional[str]:
+        """Which signal, if any, says this is a challenge page."""
         if _url_looks_blocked(url):
-            return True
+            return f"redirected to {url!r}"
         try:
             if await page.locator(_SCHOLAR_BLOCK_SELECTOR).count():
-                return True
+                return "challenge element present in the page"
         except Exception:
             pass
-        return _looks_like_scholar_block(await page.content())
+        if _looks_like_scholar_block(await page.content()):
+            return "page wording matches Google's interstitial"
+        return None
+
+    # Classify before waiting on anything. A page carrying no results at
+    # all is decidable the moment navigation completes - Scholar is
+    # server-rendered, so a genuine results page already has div.gs_ri in
+    # the DOM - and every signal (redirect URL, challenge element, Google's
+    # wording) is available now. Doing this here rather than in the timeout
+    # handler is what stops a block costing a full selector wait: measured
+    # against the deployment, classification was already correct but
+    # arrived 30s late for exactly this reason.
+    #
+    # The "no results" guard is also what makes the wording check safe this
+    # early: a real search for "unusual traffic" returns papers, so it
+    # never reaches here.
+    if await page.locator(_SCHOLAR_RESULT_SELECTOR).count() == 0:
+        early_evidence = await _block_evidence()
+        if early_evidence:
+            raise GoogleScholarBlockedException(early_evidence)
 
     try:
         await page.wait_for_selector(
             f"{_SCHOLAR_RESULT_SELECTOR}, {_SCHOLAR_BLOCK_SELECTOR}",
             state="attached", timeout=timeout_ms)
     except PlaywrightTimeoutError:
-        if await _blocked():
-            raise GoogleScholarBlockedException() from None
+        evidence = await _block_evidence()
+        if evidence:
+            raise GoogleScholarBlockedException(evidence) from None
         raise GoogleScholarUnavailableError(
             "Google Scholar returned neither results nor a recognisable "
             f"challenge page ({await _describe_page(page, url)})") from None
 
-    if await page.locator(_SCHOLAR_RESULT_SELECTOR).count() == 0 or await _blocked():
-        raise GoogleScholarBlockedException()
+    # Results win. Only a page with none of them gets classified, which
+    # keeps the wording check from firing on a legitimate search whose
+    # snippets happen to quote Google's interstitial.
+    if await page.locator(_SCHOLAR_RESULT_SELECTOR).count() == 0:
+        raise GoogleScholarBlockedException(
+            await _block_evidence() or "no results on the page")
 
     items = await page.locator(_SCHOLAR_RESULT_SELECTOR).all()
     results = []

tests/test_serp_playwright.py

@@ -283,11 +283,51 @@ async def test_a_sorry_redirect_url_counts_as_a_block(page_factory):
     """Google redirects blocked clients to /sorry/index. The final URL is
     markup-independent evidence, unlike any selector we can guess at."""
     page = await page_factory("<html><body><p>nothing recognisable here</p></body></html>")
-    page.url_override = "https://scholar.google.com/sorry/index?continue=..."
 
     with pytest.raises(GoogleScholarBlockedException):
         await _extract_google_scholar_results(
-            page, 10, timeout_ms=1000, final_url=page.url_override)
+            page, 10, timeout_ms=1000,
+            final_url="https://scholar.google.com/sorry/index?continue=...")
+
+
+async def test_a_sorry_redirect_is_recognised_without_any_waiting(page_factory):
+    """The URL is known before the page is even inspected, so a /sorry
+    redirect must not cost a selector wait.
+
+    Measured against the live deployment, this was the whole remaining
+    problem: classification was correct but arrived only after the full
+    30s timeout, because the URL was checked in the timeout handler rather
+    than up front. A long timeout is passed deliberately - the assertion
+    is that it is never reached.
+    """
+    import time
+
+    page = await page_factory("<html><body><p>nothing recognisable here</p></body></html>")
+
+    started = time.perf_counter()
+    with pytest.raises(GoogleScholarBlockedException):
+        await _extract_google_scholar_results(
+            page, 10, timeout_ms=30_000,
+            final_url="https://scholar.google.com/sorry/index?continue=...")
+    elapsed = time.perf_counter() - started
+
+    assert elapsed < 2, f"waited {elapsed:.1f}s on a URL that was known up front"
+
+
+async def test_a_block_reports_the_page_it_saw(page_factory):
+    """Diagnostics were added only to the unclassified path, which is the
+    path that did not fire in production. The block path needs them too -
+    otherwise "blocked" is an assertion the reader cannot check, and there
+    is no way to tell which signal fired.
+    """
+    page = await page_factory("<html><body><p>nothing recognisable here</p></body></html>")
+
+    with pytest.raises(GoogleScholarBlockedException) as exc_info:
+        await _extract_google_scholar_results(
+            page, 10, timeout_ms=1000,
+            final_url="https://scholar.google.com/sorry/index?continue=xyz")
+
+    assert "sorry" in str(exc_info.value), "the evidence must appear in the message"
 
 
 async def test_normal_scholar_results_are_unaffected(page_factory):
@@ -313,3 +353,42 @@ async def test_bing_result_hrefs_are_decoded(page_factory):
         "https://docs.python.org/3/library/asyncio.html",
         "https://example.org/direct",
     ]
+
+
+async def test_a_challenge_page_with_only_wording_is_also_recognised_immediately(page_factory):
+    """Don't leave the fast path depending on which signal happens to fire.
+
+    A page with no results at all, whose wording matches Google's
+    interstitial, is decidable the moment it loads - Scholar is
+    server-rendered, so a genuine results page has div.gs_ri in the DOM as
+    soon as navigation completes. Requiring "no results present" is what
+    makes the wording check safe this early: a real search for "unusual
+    traffic" returns papers, so it never reaches this branch.
+    """
+    import time
+
+    page = await page_factory(
+        "<html><body><p>Our systems have detected unusual traffic from your "
+        "computer network.</p></body></html>")
+
+    started = time.perf_counter()
+    with pytest.raises(GoogleScholarBlockedException):
+        await _extract_google_scholar_results(page, 10, timeout_ms=30_000)
+    elapsed = time.perf_counter() - started
+
+    assert elapsed < 2, f"waited {elapsed:.1f}s on a page that was decidable up front"
+
+
+async def test_a_real_results_page_mentioning_unusual_traffic_is_not_a_block(page_factory):
+    """The guard on the branch above, stated as a test: results win over
+    wording, so a legitimate search whose snippets contain the interstitial
+    phrasing is never misreported as a block."""
+    page = await page_factory(
+        '<html><body><div class="gs_ri"><h3><a href="https://example.org/p">'
+        "Detecting unusual traffic in networks</a></h3>"
+        '<div class="gs_rs">Our systems have detected unusual traffic patterns.</div>'
+        "</div></body></html>")
+
+    results = await _extract_google_scholar_results(page, 10)
+
+    assert [r["title"] for r in results] == ["Detecting unusual traffic in networks"]