Spaces:
Sleeping
Sleeping
Commit History
-
Add rate limiting, tests, better extraction, and HF auto-deploy
Claude +1: Claude Sonnet 5Unverified signatureBuilt via TDD: each module below was preceded by a failing test suite in tests/, then implemented to pass it (66 tests total). - Restructure main.py into app/ modules (config, security, rate_limit, cache, extractor, scraper, main) instead of one file. - Add a per-IP sliding-window rate limiter and an optional API key gate on /scrape, closing the abuse/cost gap on an endpoint that launches a real browser per call. - Replace the regex-based ad/nav-stripping heuristic with trafilatura, a readability-style extractor that identifies boilerplate instead of pattern-matching class/id substrings (fewer false-positive removals of legitimate content). - Add a short-TTL in-memory cache so repeat requests for the same URL skip re-rendering with a full browser. - Add .github/workflows/ci-deploy.yml: runs the test suite on every push/PR, then pushes to the Hugging Face Space's main branch (needs an HF_TOKEN repository secret) so deploys no longer require a manual git push. - Upda…ebab0e2 -
Harden scraper API: SSRF guard, browser reuse, pinned deps
Claude +1: Claude Sonnet 5Unverified signature- Reject scrape targets that resolve to private/loopback/link-local/ reserved addresses or use a non-http(s) scheme, closing an SSRF gap where any caller could point the server at internal services. - Launch a single long-lived browser via FastAPI's lifespan hook and reuse it per request (isolated per-request context) instead of spawning a fresh Chromium process on every call; bound concurrent scrapes with a semaphore so load can't exhaust the container. - Ensure the browser context is always closed, including when page setup itself fails, fixing a resource leak. - Map failures to distinct HTTP statuses (400 bad input, 502 fetch failure, 504 timeout) instead of collapsing everything into a 500 that echoed raw exception text back to the caller. - Validate the GET /scrape url the same way as POST (HttpUrl) instead of accepting an unvalidated string. - Switch networkidle -> domcontentloaded, which is less prone to false timeouts on pages with polling/analytics/websocke…cc29031
-
Upload Dockerfile
Verified signature6fe80f2 -
Upload 4 files
Verified signature70e96dc -
Upload 4 files
Verified signature5d89b25 -
Upload 4 files
Verified signature5795c69