Safer concurrent deploys, pinned dependencies, and a CONTRIBUTING guide (#5)

.github/workflows/deploy-to-hf.yml

@@ -41,4 +41,10 @@ jobs:
           HF_TOKEN: ${{ secrets.HF_TOKEN }}
         run: |
           git remote add space https://hf-actions-bot:${HF_TOKEN}@huggingface.co/spaces/OrganizedProgrammers/SERPent
-          git push --force space HEAD:main
+          # Fetch first so --force-with-lease has a remote-tracking ref to
+          # compare against - without this it either fails every run or
+          # (depending on git version) silently behaves like plain --force.
+          # This makes the push fail instead of silently overwriting anyone
+          # else who deployed to this same Space since our last fetch.
+          git fetch space main
+          git push --force-with-lease space HEAD:main

.github/workflows/test.yml

@@ -0,0 +1,33 @@
+name: Tests
+
+# Runs the suite on every pull request targeting main, so a change gets
+# feedback before merge instead of only at deploy time. deploy-to-hf.yml
+# runs its own copy of this same gate right before every push to the Space
+# (belt-and-suspenders: never trust a check that ran before the current
+# state existed, the same principle behind that workflow's
+# --force-with-lease push).
+on:
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: pip install -r requirements-dev.txt
+
+      - name: Install Playwright's Chromium
+        run: playwright install --with-deps chromium
+
+      - name: Run tests
+        run: pytest

CONTRIBUTING.md

@@ -0,0 +1,122 @@
+# Contributing to SERPent
+
+This file describes how this codebase is developed, not just how to run it.
+It exists because the project went from zero tests to a real safety net in
+one focused pass, and the point of that work is lost if the next change
+skips the same discipline. It ships as a normal file in this repo, which
+means it also lands on the Hugging Face Space on every deploy (see
+"Deployment" below) — it isn't GitHub-only documentation.
+
+## Setup
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements-dev.txt
+playwright install chromium   # only if not already present on the machine
+```
+
+Dependencies in `requirements.txt` / `requirements-dev.txt` are pinned to
+specific tested ranges, not left bare — see the comment at the top of
+`requirements.txt` for the policy and how to bump one. This matters more
+than it looks: coming back to this project after any gap in time, a bare
+`pip install` with no pins would silently grab whatever's newest on PyPI
+that day, and you'd have no way to tell "my change broke this" from "a
+dependency update broke this."
+
+## Running tests
+
+```bash
+pytest
+```
+
+97 tests, a few seconds, fully offline. All outbound HTTP is mocked with
+`respx`; the Playwright-driven scrapers (Bing, Brave, Google Scholar,
+Google Patents search) run against a real headless Chromium but navigate to
+local fixture HTML instead of the live sites — see `tests/helpers.py` for
+how and why. Nothing here depends on, or can trip, any external site's
+anti-bot measures.
+
+**Important limitation to keep in mind:** this suite protects against
+regressions in this code. It cannot catch a live site changing its markup,
+because the fixtures are frozen snapshots of the selector contract as of
+when they were written — they'll stay green forever even if Bing's HTML
+changes tomorrow. After any real gap in development, or if a scraper starts
+returning empty/wrong results in production, run a live smoke test against
+the deployed MCP server or REST API before assuming the code is at fault:
+
+```
+search(["some query"])
+scrap_patent("US10000000B2")   # or any real, known-good patent id
+```
+
+If a backend's selectors have drifted, that's a normal, expected kind of
+bug here — fix it the same way as anything else (see below): update the
+fixture HTML to match the new real markup, confirm the test fails against
+the current selectors, fix the selectors, confirm green.
+
+## The development loop (TDD, not test-after)
+
+Every change in this codebase's history followed the same loop, and new
+changes should too:
+
+1. **Write the test first.** For new behavior, write it against the
+   behavior you want and confirm it fails (red) for the right reason - not
+   a typo, an import error, or a fixture bug.
+2. **For a refactor** (behavior should stay identical), write
+   characterization tests for the *current* behavior first if none exist,
+   confirm they pass against the unchanged code, and only then refactor
+   under that safety net. `app.py` had zero tests before its endpoint
+   deduplication; the tests were added and confirmed green against the old
+   code before a single line of the refactor happened.
+3. **Implement the minimal change** to make the new/updated test pass.
+4. **Run the full suite**, not just the new test - a shared helper's bug
+   should show up as multiple failures, and that's a sign it's finding real
+   things, not a nuisance.
+5. **Distrust a test that never fails.** When a test's value is in
+   question, deliberately break the code it's supposed to protect and
+   confirm the test actually catches it, then revert. This caught a real
+   gap once already: a circuit breaker test that looked correct was
+   actually vacuous until this step exposed it.
+6. **Watch for shared mutable state across tests.** A few characterization
+   tests in this codebase turned out to be recording real state into a
+   module-level singleton (a circuit breaker) with nothing resetting it
+   between tests - harmless by luck at the time, but the kind of thing that
+   silently breaks an unrelated, later test once one more test is added.
+   Give shared singletons a fresh instance per test (an autouse fixture,
+   or explicit monkeypatching) rather than relying on the real one.
+7. **Validate risky changes for real, not just by review.** The one Docker
+   change in this project's history that could plausibly have broken the
+   live deployment (running the container as non-root, which changes
+   whether Chromium's sandbox can start at all) was verified by actually
+   building and running the image locally and confirming a real scrape
+   request completed - not by reasoning about it from the Dockerfile alone.
+
+## CI / deployment
+
+- **`.github/workflows/test.yml`** runs the full suite on every pull
+  request against `main`, so a change gets feedback before merge.
+- **`.github/workflows/deploy-to-hf.yml`** runs the same suite again
+  (`test` job) on every push to `main`, and only then (`needs: test`)
+  pushes the repo to the Hugging Face Space (`deploy` job) with
+  `git push --force-with-lease`. That flag matters: a plain `--force` push
+  is a blind overwrite with no way to tell if someone else pushed to the
+  same Space in the meantime; `--force-with-lease` fails loudly instead of
+  silently clobbering a concurrent change. If you ever see that step fail
+  with "stale info," it means exactly that - fetch, look at what's there,
+  and reconcile before retrying.
+- Because the deploy step pushes the whole repo tree (`HEAD:main`), not a
+  curated subset, everything here - including this file, the tests, and
+  the CI config - ends up on the Space too. There's nothing GitHub-only
+  about this process.
+
+## Secrets to know about
+
+These expire or rotate silently and are the most likely source of a
+deploy failure that has nothing to do with the code:
+
+- `HF_TOKEN` (repo secret) - used to push to the Hugging Face Space.
+- `OPS_CONSUMER_KEY` / `OPS_CONSUMER_SECRET` (optional, set on the Space) -
+  EPO OPS API credentials for the patent-search fallback.
+- `SERPENT_API_KEY` (optional, set on the Space) - if set, gates every
+  request behind an API key; if you set this and then can't reach the
+  deployed API, check here first.

README.md

@@ -26,3 +26,6 @@ The suite mocks all outbound HTTP (via `respx`) and drives the Playwright-based
 scrapers against local fixture HTML instead of live search engines - see
 `tests/helpers.py` for how - so it runs fully offline and doesn't depend on,
 or risk tripping, any site's anti-bot measures.
+
+See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development methodology
+(TDD workflow, dependency pinning policy, CI/deployment) behind this project.

requirements-dev.txt

@@ -1,7 +1,12 @@
 # Test-only dependencies. Install with:
 #   pip install -r requirements.txt -r requirements-dev.txt
 #   playwright install chromium   # if not already present on the machine
+#
+# See the pinning note at the top of requirements.txt - same policy here.
+# pytest-asyncio is pinned tighter (patch-only): its event-loop-scope
+# semantics have changed in ways that mattered to us before (see
+# tests/conftest.py's comment on the `browser` fixture).
 -r requirements.txt
 pytest>=8,<10
-pytest-asyncio>=0.24
-respx>=0.21
+pytest-asyncio>=1.4.0,<1.5
+respx>=0.23.1,<0.24

requirements.txt

@@ -1,10 +1,24 @@
-fastapi
-uvicorn
-pydantic
-playwright
-duckduckgo_search
-beautifulsoup4
-httpx
-lxml
-python-dotenv
-fastmcp>=3.0,<4
+# Pinned to the versions this codebase is actually tested against (see
+# CONTRIBUTING.md). Range = "current version, up to the next breaking
+# boundary" - the next major for a stable (>=1.0) package, the next minor
+# for a 0.x package (0.x minor bumps are semver's equivalent of a major).
+#
+# playwright and duckduckgo_search are pinned tighter (patch-only): playwright
+# ships in lockstep with a specific browser revision (a version mismatch is a
+# real, previously-hit failure mode - see tests/conftest.py's fallback
+# executable_path logic), and duckduckgo_search has a history of breaking
+# changes between versions.
+#
+# To bump a dependency: update the pin here, run `pytest`, and if you touch
+# playwright specifically, re-run the full suite including
+# tests/test_serp_playwright.py (it drives a real browser).
+fastapi>=0.141.1,<0.142
+uvicorn>=0.52.4,<0.53
+pydantic>=2.13.5,<3
+playwright>=1.62.0,<1.63
+duckduckgo_search>=8.1.1,<8.2
+beautifulsoup4>=4.15.0,<5
+httpx>=0.28.1,<0.29
+lxml>=6.1.2,<7
+python-dotenv>=1.2.3,<2
+fastmcp>=3.4.7,<4