Spaces:
Paused
Paused
Peng Ding commited on
Commit Β·
9178807
1
Parent(s): a4ba91f
docs: update changelog and CLAUDE.md for worker pool + Patchright
Browse files- Changelog: add all unreleased changes (worker pool, Patchright,
Docker optimization, i18n, metrics, auto-download binary)
- CLAUDE.md: update architecture description, add deployment modes,
environment variables, new commands, updated repo layout
- CHANGELOG.md +19 -0
- CLAUDE.md +50 -14
CHANGELOG.md
CHANGED
|
@@ -9,14 +9,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
| 9 |
|
| 10 |
### Added
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
- Prometheus metrics endpoint at `GET /metrics` β zero-dependency exposition format with uptime, browser status, active pages, request counters, cache lookups, and latency summaries (p50/p95) ([#23])
|
| 13 |
- Dashboard i18n: language selector dropdown (en/zh) with localStorage persistence; extensible via dict β adding a language only requires a new entry ([#22])
|
| 14 |
- Dashboard `GET /stats` JSON API for live data polling ([#22])
|
| 15 |
- SVG ring gauge for capacity visualization on dashboard ([#22])
|
| 16 |
- Author credit and badges in dashboard footer ([#22])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
### Changed
|
| 19 |
|
|
|
|
|
|
|
|
|
|
| 20 |
- Dashboard redesign: dark theme inspired by modal.com β pure black background, mint-green accents, DM Sans + JetBrains Mono typography ([#22])
|
| 21 |
- Dashboard data refresh: replaced `<meta http-equiv="refresh">` full-page reload with fetch-based polling β no more flashing or scroll reset ([#22])
|
| 22 |
- Dashboard shows stale-data indicator (dims) after 3 consecutive fetch failures ([#22])
|
|
@@ -24,10 +40,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
| 24 |
- Shields.io badges use unified dark-green theme colors ([#22])
|
| 25 |
- Vendored modules: replace deprecated `jsonc` with `jsonx`, remove `benchmark_compare` ([#22])
|
| 26 |
- Bump vendored config 0.3.0β0.3.1, httpserver 0.1.0β0.2.1, readability 0.1.0β0.2.0
|
|
|
|
| 27 |
|
| 28 |
### Removed
|
| 29 |
|
| 30 |
- HF Spaces badge (space was taken down) ([#22])
|
|
|
|
| 31 |
|
| 32 |
## [0.3.1] - 2026-07-04
|
| 33 |
|
|
@@ -134,3 +152,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
| 134 |
[#21]: https://github.com/Oaklight/veilrender/pull/21
|
| 135 |
[#22]: https://github.com/Oaklight/veilrender/pull/22
|
| 136 |
[#23]: https://github.com/Oaklight/veilrender/pull/23
|
|
|
|
|
|
| 9 |
|
| 10 |
### Added
|
| 11 |
|
| 12 |
+
- **Remote browser worker pool** for horizontal scaling β `VEILRENDER_WORKERS` env var routes requests to remote CDP endpoints via least-connections routing ([#25])
|
| 13 |
+
- `LocalWorker` / `RemoteWorker` abstraction with shared `_BaseWorker` interface ([#25])
|
| 14 |
+
- Worker health checks with automatic reconnection for remote workers ([#25])
|
| 15 |
+
- CDP proxy `?worker=N` query param for targeted worker routing ([#25])
|
| 16 |
+
- Per-worker Prometheus gauges: `veilrender_worker_healthy`, `veilrender_worker_active_pages`, `veilrender_worker_browser_pages` ([#25])
|
| 17 |
+
- Real browser page count via CDP `/json` endpoint for accurate capacity reporting ([#25])
|
| 18 |
- Prometheus metrics endpoint at `GET /metrics` β zero-dependency exposition format with uptime, browser status, active pages, request counters, cache lookups, and latency summaries (p50/p95) ([#23])
|
| 19 |
- Dashboard i18n: language selector dropdown (en/zh) with localStorage persistence; extensible via dict β adding a language only requires a new entry ([#22])
|
| 20 |
- Dashboard `GET /stats` JSON API for live data polling ([#22])
|
| 21 |
- SVG ring gauge for capacity visualization on dashboard ([#22])
|
| 22 |
- Author credit and badges in dashboard footer ([#22])
|
| 23 |
+
- Multi-target Dockerfile: `gateway` (336MB, no browser) and `full` (1.07GB, CloakBrowser embedded)
|
| 24 |
+
- `deploy/compose.yaml` (single-instance) and `deploy/compose-pool.yaml` (gateway + `cloakhq/cloakbrowser` workers)
|
| 25 |
+
- `POOL=1` flag for `make deploy-dev` to deploy in pool mode
|
| 26 |
+
- `CLOAKBROWSER_MIRROR` env var for China downloads (e.g. `https://ghfast.top`)
|
| 27 |
+
- `CLOAKBROWSER_BINARY` env var to use a custom browser binary path
|
| 28 |
+
- Auto-download CloakBrowser binary from GitHub Releases without pip package
|
| 29 |
+
- `scripts/download-cloakbrowser.py` for standalone binary download
|
| 30 |
|
| 31 |
### Changed
|
| 32 |
|
| 33 |
+
- **Migrate from Playwright to Patchright** β drop-in replacement with stealth driver patches for Chromium ([#25])
|
| 34 |
+
- Remove `cloakbrowser` pip dependency β binary downloaded directly from GitHub Releases, stealth args generated inline
|
| 35 |
+
- Docker image size: full 1.43GB β 1.07GB, gateway 498MB β 336MB (removed playwright transitive dep + multi-stage libgbm extraction)
|
| 36 |
- Dashboard redesign: dark theme inspired by modal.com β pure black background, mint-green accents, DM Sans + JetBrains Mono typography ([#22])
|
| 37 |
- Dashboard data refresh: replaced `<meta http-equiv="refresh">` full-page reload with fetch-based polling β no more flashing or scroll reset ([#22])
|
| 38 |
- Dashboard shows stale-data indicator (dims) after 3 consecutive fetch failures ([#22])
|
|
|
|
| 40 |
- Shields.io badges use unified dark-green theme colors ([#22])
|
| 41 |
- Vendored modules: replace deprecated `jsonc` with `jsonx`, remove `benchmark_compare` ([#22])
|
| 42 |
- Bump vendored config 0.3.0β0.3.1, httpserver 0.1.0β0.2.1, readability 0.1.0β0.2.0
|
| 43 |
+
- New config: `VEILRENDER_WORKERS`, `VEILRENDER_WORKER_MAX_CONCURRENT`, `VEILRENDER_WORKER_HEALTH_INTERVAL` ([#25])
|
| 44 |
|
| 45 |
### Removed
|
| 46 |
|
| 47 |
- HF Spaces badge (space was taken down) ([#22])
|
| 48 |
+
- `cloakbrowser` pip dependency (binary auto-downloaded instead)
|
| 49 |
|
| 50 |
## [0.3.1] - 2026-07-04
|
| 51 |
|
|
|
|
| 152 |
[#21]: https://github.com/Oaklight/veilrender/pull/21
|
| 153 |
[#22]: https://github.com/Oaklight/veilrender/pull/22
|
| 154 |
[#23]: https://github.com/Oaklight/veilrender/pull/23
|
| 155 |
+
[#25]: https://github.com/Oaklight/veilrender/pull/25
|
CLAUDE.md
CHANGED
|
@@ -2,18 +2,21 @@
|
|
| 2 |
|
| 3 |
## What this project is
|
| 4 |
|
| 5 |
-
Headless browser rendering API. Accepts a URL, renders it with
|
| 6 |
-
Chromium, returns HTML/Markdown/readability content.
|
| 7 |
-
|
| 8 |
|
| 9 |
## Architecture
|
| 10 |
|
| 11 |
-
- **Zero external deps** except `
|
| 12 |
in `src/veilrender/_vendor/` for HTTP server, HTML parsing, readability, etc.
|
| 13 |
- **`httpserver`** (zerodep) as the async HTTP framework β not FastAPI
|
| 14 |
-
- **
|
| 15 |
-
|
| 16 |
- **Stateless** β no session management, no database
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
## Repository layout
|
| 19 |
|
|
@@ -22,23 +25,55 @@ src/veilrender/
|
|
| 22 |
βββ app.py # HTTP server setup, route registration, main()
|
| 23 |
βββ config.py # Settings from env vars (VEILRENDER_ prefix)
|
| 24 |
βββ auth.py # Token verification
|
| 25 |
-
βββ browser.py #
|
|
|
|
| 26 |
βββ models.py # Request/response dataclasses
|
| 27 |
-
βββ routes/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
βββ _vendor/ # zerodep modules β DO NOT EDIT manually
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
```
|
| 30 |
|
| 31 |
## Commands
|
| 32 |
|
| 33 |
```bash
|
| 34 |
-
make dev
|
| 35 |
-
make build
|
| 36 |
-
make
|
| 37 |
-
make
|
| 38 |
-
make
|
| 39 |
-
make
|
|
|
|
|
|
|
|
|
|
| 40 |
```
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
## Files to never edit
|
| 43 |
|
| 44 |
- `src/veilrender/_vendor/**` β vendored zerodep modules, update via
|
|
@@ -49,3 +84,4 @@ make vendor # Re-vendor zerodep modules
|
|
| 49 |
1. `ruff check --fix && ruff format` on changed Python files
|
| 50 |
2. Server starts and `/health` returns 200
|
| 51 |
3. `/render` returns valid content for a test URL
|
|
|
|
|
|
| 2 |
|
| 3 |
## What this project is
|
| 4 |
|
| 5 |
+
Headless browser rendering API. Accepts a URL, renders it with Patchright +
|
| 6 |
+
CloakBrowser (stealth Chromium), returns HTML/Markdown/readability content.
|
| 7 |
+
Supports single-instance and gateway+worker pool deployment modes.
|
| 8 |
|
| 9 |
## Architecture
|
| 10 |
|
| 11 |
+
- **Zero external deps** except `patchright` β uses vendored `zerodep` modules
|
| 12 |
in `src/veilrender/_vendor/` for HTTP server, HTML parsing, readability, etc.
|
| 13 |
- **`httpserver`** (zerodep) as the async HTTP framework β not FastAPI
|
| 14 |
+
- **Patchright** (stealth Playwright fork) for browser control via CDP
|
| 15 |
+
- **CloakBrowser** stealth Chromium binary β auto-downloaded, no pip package needed
|
| 16 |
- **Stateless** β no session management, no database
|
| 17 |
+
- **Two deployment modes**:
|
| 18 |
+
- **Full** (single-instance): embedded CloakBrowser, 1.07GB image
|
| 19 |
+
- **Gateway + Workers** (pool): gateway routes to remote `cloakhq/cloakbrowser` CDP workers, 336MB gateway image
|
| 20 |
|
| 21 |
## Repository layout
|
| 22 |
|
|
|
|
| 25 |
βββ app.py # HTTP server setup, route registration, main()
|
| 26 |
βββ config.py # Settings from env vars (VEILRENDER_ prefix)
|
| 27 |
βββ auth.py # Token verification
|
| 28 |
+
βββ browser.py # Browser lifecycle: LocalWorker, RemoteWorker, BrowserManager pool
|
| 29 |
+
βββ filters.py # Outbound request blocklist filtering
|
| 30 |
βββ models.py # Request/response dataclasses
|
| 31 |
+
βββ routes/
|
| 32 |
+
β βββ dashboard.py # GET / β stats dashboard with i18n
|
| 33 |
+
β βββ health.py # GET /health β liveness check
|
| 34 |
+
β βββ metrics.py # GET /metrics β Prometheus exposition format
|
| 35 |
+
β βββ render.py # POST /render β URL β HTML/Markdown/readability
|
| 36 |
+
β βββ screenshot.py # POST /screenshot β URL β PNG
|
| 37 |
+
βββ stats.py # In-memory request counters and latency tracking
|
| 38 |
βββ _vendor/ # zerodep modules β DO NOT EDIT manually
|
| 39 |
+
deploy/
|
| 40 |
+
βββ compose.yaml # Single-instance Docker Compose
|
| 41 |
+
βββ compose-pool.yaml # Gateway + CloakBrowser worker pool
|
| 42 |
+
scripts/
|
| 43 |
+
βββ download-cloakbrowser.py # Download CloakBrowser binary (no pip needed)
|
| 44 |
```
|
| 45 |
|
| 46 |
## Commands
|
| 47 |
|
| 48 |
```bash
|
| 49 |
+
make dev # Run dev server on :7860
|
| 50 |
+
make build # Docker build (full, with CloakBrowser)
|
| 51 |
+
make build-gateway # Docker build (gateway only, no browser)
|
| 52 |
+
make run # Docker run
|
| 53 |
+
make lint # ruff check --fix && ruff format
|
| 54 |
+
make typecheck # ty check
|
| 55 |
+
make vendor # Re-vendor zerodep modules
|
| 56 |
+
make deploy-dev SSH_TARGET=host # Deploy single-instance
|
| 57 |
+
make deploy-dev SSH_TARGET=host POOL=1 # Deploy gateway + worker pool
|
| 58 |
```
|
| 59 |
|
| 60 |
+
## Environment variables
|
| 61 |
+
|
| 62 |
+
### Core
|
| 63 |
+
- `VEILRENDER_API_TOKEN` β Bearer token for auth
|
| 64 |
+
- `VEILRENDER_PORT` β Listen port (default 7860)
|
| 65 |
+
- `VEILRENDER_MAX_CONCURRENT` β Max concurrent pages (default 5)
|
| 66 |
+
- `VEILRENDER_TIMEOUT` β Page load timeout in ms (default 30000)
|
| 67 |
+
|
| 68 |
+
### Worker pool
|
| 69 |
+
- `VEILRENDER_WORKERS` β Comma-separated remote CDP endpoints (enables pool mode)
|
| 70 |
+
- `VEILRENDER_WORKER_MAX_CONCURRENT` β Per-worker concurrency limit
|
| 71 |
+
- `VEILRENDER_WORKER_HEALTH_INTERVAL` β Health check interval in seconds (default 10)
|
| 72 |
+
|
| 73 |
+
### Browser binary
|
| 74 |
+
- `CLOAKBROWSER_BINARY` β Path to custom browser binary
|
| 75 |
+
- `CLOAKBROWSER_MIRROR` β GitHub mirror for China downloads (e.g. `https://ghfast.top`)
|
| 76 |
+
|
| 77 |
## Files to never edit
|
| 78 |
|
| 79 |
- `src/veilrender/_vendor/**` β vendored zerodep modules, update via
|
|
|
|
| 84 |
1. `ruff check --fix && ruff format` on changed Python files
|
| 85 |
2. Server starts and `/health` returns 200
|
| 86 |
3. `/render` returns valid content for a test URL
|
| 87 |
+
4. `/metrics` returns valid Prometheus output
|