Spaces:
Sleeping
Document the development methodology (CONTRIBUTING.md)
Browse filesWrites down the TDD workflow this project has followed throughout -
test/characterize first, run the full suite (a shared helper's bug should
surface as multiple failures), distrust a test that never fails (break the
code on purpose and confirm it catches it), watch for shared mutable state
leaking across tests, and validate genuinely risky changes for real rather
than by review alone - plus the CI/deploy pipeline, the dependency pinning
policy, and which secrets to check first when something breaks for no
obvious code reason.
This isn't GitHub-only documentation: deploy-to-hf.yml pushes the whole
repo tree to the Hugging Face Space on every deploy, not a curated subset,
so this file (like the tests, the pinned requirements, and the CI config)
ships there too once this merges and deploys.
Linked from README.md.
- CONTRIBUTING.md +122 -0
- README.md +3 -0
|
@@ -0,0 +1,122 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Contributing to SERPent
|
| 2 |
+
|
| 3 |
+
This file describes how this codebase is developed, not just how to run it.
|
| 4 |
+
It exists because the project went from zero tests to a real safety net in
|
| 5 |
+
one focused pass, and the point of that work is lost if the next change
|
| 6 |
+
skips the same discipline. It ships as a normal file in this repo, which
|
| 7 |
+
means it also lands on the Hugging Face Space on every deploy (see
|
| 8 |
+
"Deployment" below) β it isn't GitHub-only documentation.
|
| 9 |
+
|
| 10 |
+
## Setup
|
| 11 |
+
|
| 12 |
+
```bash
|
| 13 |
+
python3 -m venv .venv && source .venv/bin/activate
|
| 14 |
+
pip install -r requirements-dev.txt
|
| 15 |
+
playwright install chromium # only if not already present on the machine
|
| 16 |
+
```
|
| 17 |
+
|
| 18 |
+
Dependencies in `requirements.txt` / `requirements-dev.txt` are pinned to
|
| 19 |
+
specific tested ranges, not left bare β see the comment at the top of
|
| 20 |
+
`requirements.txt` for the policy and how to bump one. This matters more
|
| 21 |
+
than it looks: coming back to this project after any gap in time, a bare
|
| 22 |
+
`pip install` with no pins would silently grab whatever's newest on PyPI
|
| 23 |
+
that day, and you'd have no way to tell "my change broke this" from "a
|
| 24 |
+
dependency update broke this."
|
| 25 |
+
|
| 26 |
+
## Running tests
|
| 27 |
+
|
| 28 |
+
```bash
|
| 29 |
+
pytest
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
97 tests, a few seconds, fully offline. All outbound HTTP is mocked with
|
| 33 |
+
`respx`; the Playwright-driven scrapers (Bing, Brave, Google Scholar,
|
| 34 |
+
Google Patents search) run against a real headless Chromium but navigate to
|
| 35 |
+
local fixture HTML instead of the live sites β see `tests/helpers.py` for
|
| 36 |
+
how and why. Nothing here depends on, or can trip, any external site's
|
| 37 |
+
anti-bot measures.
|
| 38 |
+
|
| 39 |
+
**Important limitation to keep in mind:** this suite protects against
|
| 40 |
+
regressions in this code. It cannot catch a live site changing its markup,
|
| 41 |
+
because the fixtures are frozen snapshots of the selector contract as of
|
| 42 |
+
when they were written β they'll stay green forever even if Bing's HTML
|
| 43 |
+
changes tomorrow. After any real gap in development, or if a scraper starts
|
| 44 |
+
returning empty/wrong results in production, run a live smoke test against
|
| 45 |
+
the deployed MCP server or REST API before assuming the code is at fault:
|
| 46 |
+
|
| 47 |
+
```
|
| 48 |
+
search(["some query"])
|
| 49 |
+
scrap_patent("US10000000B2") # or any real, known-good patent id
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
If a backend's selectors have drifted, that's a normal, expected kind of
|
| 53 |
+
bug here β fix it the same way as anything else (see below): update the
|
| 54 |
+
fixture HTML to match the new real markup, confirm the test fails against
|
| 55 |
+
the current selectors, fix the selectors, confirm green.
|
| 56 |
+
|
| 57 |
+
## The development loop (TDD, not test-after)
|
| 58 |
+
|
| 59 |
+
Every change in this codebase's history followed the same loop, and new
|
| 60 |
+
changes should too:
|
| 61 |
+
|
| 62 |
+
1. **Write the test first.** For new behavior, write it against the
|
| 63 |
+
behavior you want and confirm it fails (red) for the right reason - not
|
| 64 |
+
a typo, an import error, or a fixture bug.
|
| 65 |
+
2. **For a refactor** (behavior should stay identical), write
|
| 66 |
+
characterization tests for the *current* behavior first if none exist,
|
| 67 |
+
confirm they pass against the unchanged code, and only then refactor
|
| 68 |
+
under that safety net. `app.py` had zero tests before its endpoint
|
| 69 |
+
deduplication; the tests were added and confirmed green against the old
|
| 70 |
+
code before a single line of the refactor happened.
|
| 71 |
+
3. **Implement the minimal change** to make the new/updated test pass.
|
| 72 |
+
4. **Run the full suite**, not just the new test - a shared helper's bug
|
| 73 |
+
should show up as multiple failures, and that's a sign it's finding real
|
| 74 |
+
things, not a nuisance.
|
| 75 |
+
5. **Distrust a test that never fails.** When a test's value is in
|
| 76 |
+
question, deliberately break the code it's supposed to protect and
|
| 77 |
+
confirm the test actually catches it, then revert. This caught a real
|
| 78 |
+
gap once already: a circuit breaker test that looked correct was
|
| 79 |
+
actually vacuous until this step exposed it.
|
| 80 |
+
6. **Watch for shared mutable state across tests.** A few characterization
|
| 81 |
+
tests in this codebase turned out to be recording real state into a
|
| 82 |
+
module-level singleton (a circuit breaker) with nothing resetting it
|
| 83 |
+
between tests - harmless by luck at the time, but the kind of thing that
|
| 84 |
+
silently breaks an unrelated, later test once one more test is added.
|
| 85 |
+
Give shared singletons a fresh instance per test (an autouse fixture,
|
| 86 |
+
or explicit monkeypatching) rather than relying on the real one.
|
| 87 |
+
7. **Validate risky changes for real, not just by review.** The one Docker
|
| 88 |
+
change in this project's history that could plausibly have broken the
|
| 89 |
+
live deployment (running the container as non-root, which changes
|
| 90 |
+
whether Chromium's sandbox can start at all) was verified by actually
|
| 91 |
+
building and running the image locally and confirming a real scrape
|
| 92 |
+
request completed - not by reasoning about it from the Dockerfile alone.
|
| 93 |
+
|
| 94 |
+
## CI / deployment
|
| 95 |
+
|
| 96 |
+
- **`.github/workflows/test.yml`** runs the full suite on every pull
|
| 97 |
+
request against `main`, so a change gets feedback before merge.
|
| 98 |
+
- **`.github/workflows/deploy-to-hf.yml`** runs the same suite again
|
| 99 |
+
(`test` job) on every push to `main`, and only then (`needs: test`)
|
| 100 |
+
pushes the repo to the Hugging Face Space (`deploy` job) with
|
| 101 |
+
`git push --force-with-lease`. That flag matters: a plain `--force` push
|
| 102 |
+
is a blind overwrite with no way to tell if someone else pushed to the
|
| 103 |
+
same Space in the meantime; `--force-with-lease` fails loudly instead of
|
| 104 |
+
silently clobbering a concurrent change. If you ever see that step fail
|
| 105 |
+
with "stale info," it means exactly that - fetch, look at what's there,
|
| 106 |
+
and reconcile before retrying.
|
| 107 |
+
- Because the deploy step pushes the whole repo tree (`HEAD:main`), not a
|
| 108 |
+
curated subset, everything here - including this file, the tests, and
|
| 109 |
+
the CI config - ends up on the Space too. There's nothing GitHub-only
|
| 110 |
+
about this process.
|
| 111 |
+
|
| 112 |
+
## Secrets to know about
|
| 113 |
+
|
| 114 |
+
These expire or rotate silently and are the most likely source of a
|
| 115 |
+
deploy failure that has nothing to do with the code:
|
| 116 |
+
|
| 117 |
+
- `HF_TOKEN` (repo secret) - used to push to the Hugging Face Space.
|
| 118 |
+
- `OPS_CONSUMER_KEY` / `OPS_CONSUMER_SECRET` (optional, set on the Space) -
|
| 119 |
+
EPO OPS API credentials for the patent-search fallback.
|
| 120 |
+
- `SERPENT_API_KEY` (optional, set on the Space) - if set, gates every
|
| 121 |
+
request behind an API key; if you set this and then can't reach the
|
| 122 |
+
deployed API, check here first.
|
|
@@ -26,3 +26,6 @@ The suite mocks all outbound HTTP (via `respx`) and drives the Playwright-based
|
|
| 26 |
scrapers against local fixture HTML instead of live search engines - see
|
| 27 |
`tests/helpers.py` for how - so it runs fully offline and doesn't depend on,
|
| 28 |
or risk tripping, any site's anti-bot measures.
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
scrapers against local fixture HTML instead of live search engines - see
|
| 27 |
`tests/helpers.py` for how - so it runs fully offline and doesn't depend on,
|
| 28 |
or risk tripping, any site's anti-bot measures.
|
| 29 |
+
|
| 30 |
+
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the development methodology
|
| 31 |
+
(TDD workflow, dependency pinning policy, CI/deployment) behind this project.
|