Peng Ding commited on
Commit
9178807
Β·
1 Parent(s): a4ba91f

docs: update changelog and CLAUDE.md for worker pool + Patchright

Browse files

- Changelog: add all unreleased changes (worker pool, Patchright,
Docker optimization, i18n, metrics, auto-download binary)
- CLAUDE.md: update architecture description, add deployment modes,
environment variables, new commands, updated repo layout

Files changed (2) hide show
  1. CHANGELOG.md +19 -0
  2. CLAUDE.md +50 -14
CHANGELOG.md CHANGED
@@ -9,14 +9,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
9
 
10
  ### Added
11
 
 
 
 
 
 
 
12
  - Prometheus metrics endpoint at `GET /metrics` β€” zero-dependency exposition format with uptime, browser status, active pages, request counters, cache lookups, and latency summaries (p50/p95) ([#23])
13
  - Dashboard i18n: language selector dropdown (en/zh) with localStorage persistence; extensible via dict β€” adding a language only requires a new entry ([#22])
14
  - Dashboard `GET /stats` JSON API for live data polling ([#22])
15
  - SVG ring gauge for capacity visualization on dashboard ([#22])
16
  - Author credit and badges in dashboard footer ([#22])
 
 
 
 
 
 
 
17
 
18
  ### Changed
19
 
 
 
 
20
  - Dashboard redesign: dark theme inspired by modal.com β€” pure black background, mint-green accents, DM Sans + JetBrains Mono typography ([#22])
21
  - Dashboard data refresh: replaced `<meta http-equiv="refresh">` full-page reload with fetch-based polling β€” no more flashing or scroll reset ([#22])
22
  - Dashboard shows stale-data indicator (dims) after 3 consecutive fetch failures ([#22])
@@ -24,10 +40,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
24
  - Shields.io badges use unified dark-green theme colors ([#22])
25
  - Vendored modules: replace deprecated `jsonc` with `jsonx`, remove `benchmark_compare` ([#22])
26
  - Bump vendored config 0.3.0β†’0.3.1, httpserver 0.1.0β†’0.2.1, readability 0.1.0β†’0.2.0
 
27
 
28
  ### Removed
29
 
30
  - HF Spaces badge (space was taken down) ([#22])
 
31
 
32
  ## [0.3.1] - 2026-07-04
33
 
@@ -134,3 +152,4 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
134
  [#21]: https://github.com/Oaklight/veilrender/pull/21
135
  [#22]: https://github.com/Oaklight/veilrender/pull/22
136
  [#23]: https://github.com/Oaklight/veilrender/pull/23
 
 
9
 
10
  ### Added
11
 
12
+ - **Remote browser worker pool** for horizontal scaling β€” `VEILRENDER_WORKERS` env var routes requests to remote CDP endpoints via least-connections routing ([#25])
13
+ - `LocalWorker` / `RemoteWorker` abstraction with shared `_BaseWorker` interface ([#25])
14
+ - Worker health checks with automatic reconnection for remote workers ([#25])
15
+ - CDP proxy `?worker=N` query param for targeted worker routing ([#25])
16
+ - Per-worker Prometheus gauges: `veilrender_worker_healthy`, `veilrender_worker_active_pages`, `veilrender_worker_browser_pages` ([#25])
17
+ - Real browser page count via CDP `/json` endpoint for accurate capacity reporting ([#25])
18
  - Prometheus metrics endpoint at `GET /metrics` β€” zero-dependency exposition format with uptime, browser status, active pages, request counters, cache lookups, and latency summaries (p50/p95) ([#23])
19
  - Dashboard i18n: language selector dropdown (en/zh) with localStorage persistence; extensible via dict β€” adding a language only requires a new entry ([#22])
20
  - Dashboard `GET /stats` JSON API for live data polling ([#22])
21
  - SVG ring gauge for capacity visualization on dashboard ([#22])
22
  - Author credit and badges in dashboard footer ([#22])
23
+ - Multi-target Dockerfile: `gateway` (336MB, no browser) and `full` (1.07GB, CloakBrowser embedded)
24
+ - `deploy/compose.yaml` (single-instance) and `deploy/compose-pool.yaml` (gateway + `cloakhq/cloakbrowser` workers)
25
+ - `POOL=1` flag for `make deploy-dev` to deploy in pool mode
26
+ - `CLOAKBROWSER_MIRROR` env var for China downloads (e.g. `https://ghfast.top`)
27
+ - `CLOAKBROWSER_BINARY` env var to use a custom browser binary path
28
+ - Auto-download CloakBrowser binary from GitHub Releases without pip package
29
+ - `scripts/download-cloakbrowser.py` for standalone binary download
30
 
31
  ### Changed
32
 
33
+ - **Migrate from Playwright to Patchright** β€” drop-in replacement with stealth driver patches for Chromium ([#25])
34
+ - Remove `cloakbrowser` pip dependency β€” binary downloaded directly from GitHub Releases, stealth args generated inline
35
+ - Docker image size: full 1.43GB β†’ 1.07GB, gateway 498MB β†’ 336MB (removed playwright transitive dep + multi-stage libgbm extraction)
36
  - Dashboard redesign: dark theme inspired by modal.com β€” pure black background, mint-green accents, DM Sans + JetBrains Mono typography ([#22])
37
  - Dashboard data refresh: replaced `<meta http-equiv="refresh">` full-page reload with fetch-based polling β€” no more flashing or scroll reset ([#22])
38
  - Dashboard shows stale-data indicator (dims) after 3 consecutive fetch failures ([#22])
 
40
  - Shields.io badges use unified dark-green theme colors ([#22])
41
  - Vendored modules: replace deprecated `jsonc` with `jsonx`, remove `benchmark_compare` ([#22])
42
  - Bump vendored config 0.3.0β†’0.3.1, httpserver 0.1.0β†’0.2.1, readability 0.1.0β†’0.2.0
43
+ - New config: `VEILRENDER_WORKERS`, `VEILRENDER_WORKER_MAX_CONCURRENT`, `VEILRENDER_WORKER_HEALTH_INTERVAL` ([#25])
44
 
45
  ### Removed
46
 
47
  - HF Spaces badge (space was taken down) ([#22])
48
+ - `cloakbrowser` pip dependency (binary auto-downloaded instead)
49
 
50
  ## [0.3.1] - 2026-07-04
51
 
 
152
  [#21]: https://github.com/Oaklight/veilrender/pull/21
153
  [#22]: https://github.com/Oaklight/veilrender/pull/22
154
  [#23]: https://github.com/Oaklight/veilrender/pull/23
155
+ [#25]: https://github.com/Oaklight/veilrender/pull/25
CLAUDE.md CHANGED
@@ -2,18 +2,21 @@
2
 
3
  ## What this project is
4
 
5
- Headless browser rendering API. Accepts a URL, renders it with Playwright +
6
- Chromium, returns HTML/Markdown/readability content. Primary deployment target:
7
- HF Spaces (free tier). Also runs as standalone Docker container.
8
 
9
  ## Architecture
10
 
11
- - **Zero external deps** except `playwright` β€” uses vendored `zerodep` modules
12
  in `src/veilrender/_vendor/` for HTTP server, HTML parsing, readability, etc.
13
  - **`httpserver`** (zerodep) as the async HTTP framework β€” not FastAPI
14
- - **Playwright** for browser control β€” shared browser instance, per-request
15
- `BrowserContext` isolation
16
  - **Stateless** β€” no session management, no database
 
 
 
17
 
18
  ## Repository layout
19
 
@@ -22,23 +25,55 @@ src/veilrender/
22
  β”œβ”€β”€ app.py # HTTP server setup, route registration, main()
23
  β”œβ”€β”€ config.py # Settings from env vars (VEILRENDER_ prefix)
24
  β”œβ”€β”€ auth.py # Token verification
25
- β”œβ”€β”€ browser.py # Playwright browser lifecycle
 
26
  β”œβ”€β”€ models.py # Request/response dataclasses
27
- β”œβ”€β”€ routes/ # Route handlers (render, screenshot, health)
 
 
 
 
 
 
28
  └── _vendor/ # zerodep modules β€” DO NOT EDIT manually
 
 
 
 
 
29
  ```
30
 
31
  ## Commands
32
 
33
  ```bash
34
- make dev # Run dev server on :7860
35
- make build # Docker build
36
- make run # Docker run
37
- make lint # ruff check --fix && ruff format
38
- make typecheck # ty check
39
- make vendor # Re-vendor zerodep modules
 
 
 
40
  ```
41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  ## Files to never edit
43
 
44
  - `src/veilrender/_vendor/**` β€” vendored zerodep modules, update via
@@ -49,3 +84,4 @@ make vendor # Re-vendor zerodep modules
49
  1. `ruff check --fix && ruff format` on changed Python files
50
  2. Server starts and `/health` returns 200
51
  3. `/render` returns valid content for a test URL
 
 
2
 
3
  ## What this project is
4
 
5
+ Headless browser rendering API. Accepts a URL, renders it with Patchright +
6
+ CloakBrowser (stealth Chromium), returns HTML/Markdown/readability content.
7
+ Supports single-instance and gateway+worker pool deployment modes.
8
 
9
  ## Architecture
10
 
11
+ - **Zero external deps** except `patchright` β€” uses vendored `zerodep` modules
12
  in `src/veilrender/_vendor/` for HTTP server, HTML parsing, readability, etc.
13
  - **`httpserver`** (zerodep) as the async HTTP framework β€” not FastAPI
14
+ - **Patchright** (stealth Playwright fork) for browser control via CDP
15
+ - **CloakBrowser** stealth Chromium binary β€” auto-downloaded, no pip package needed
16
  - **Stateless** β€” no session management, no database
17
+ - **Two deployment modes**:
18
+ - **Full** (single-instance): embedded CloakBrowser, 1.07GB image
19
+ - **Gateway + Workers** (pool): gateway routes to remote `cloakhq/cloakbrowser` CDP workers, 336MB gateway image
20
 
21
  ## Repository layout
22
 
 
25
  β”œβ”€β”€ app.py # HTTP server setup, route registration, main()
26
  β”œβ”€β”€ config.py # Settings from env vars (VEILRENDER_ prefix)
27
  β”œβ”€β”€ auth.py # Token verification
28
+ β”œβ”€β”€ browser.py # Browser lifecycle: LocalWorker, RemoteWorker, BrowserManager pool
29
+ β”œβ”€β”€ filters.py # Outbound request blocklist filtering
30
  β”œβ”€β”€ models.py # Request/response dataclasses
31
+ β”œβ”€β”€ routes/
32
+ β”‚ β”œβ”€β”€ dashboard.py # GET / β€” stats dashboard with i18n
33
+ β”‚ β”œβ”€β”€ health.py # GET /health β€” liveness check
34
+ β”‚ β”œβ”€β”€ metrics.py # GET /metrics β€” Prometheus exposition format
35
+ β”‚ β”œβ”€β”€ render.py # POST /render β€” URL β†’ HTML/Markdown/readability
36
+ β”‚ └── screenshot.py # POST /screenshot β€” URL β†’ PNG
37
+ β”œβ”€β”€ stats.py # In-memory request counters and latency tracking
38
  └── _vendor/ # zerodep modules β€” DO NOT EDIT manually
39
+ deploy/
40
+ β”œβ”€β”€ compose.yaml # Single-instance Docker Compose
41
+ └── compose-pool.yaml # Gateway + CloakBrowser worker pool
42
+ scripts/
43
+ └── download-cloakbrowser.py # Download CloakBrowser binary (no pip needed)
44
  ```
45
 
46
  ## Commands
47
 
48
  ```bash
49
+ make dev # Run dev server on :7860
50
+ make build # Docker build (full, with CloakBrowser)
51
+ make build-gateway # Docker build (gateway only, no browser)
52
+ make run # Docker run
53
+ make lint # ruff check --fix && ruff format
54
+ make typecheck # ty check
55
+ make vendor # Re-vendor zerodep modules
56
+ make deploy-dev SSH_TARGET=host # Deploy single-instance
57
+ make deploy-dev SSH_TARGET=host POOL=1 # Deploy gateway + worker pool
58
  ```
59
 
60
+ ## Environment variables
61
+
62
+ ### Core
63
+ - `VEILRENDER_API_TOKEN` β€” Bearer token for auth
64
+ - `VEILRENDER_PORT` β€” Listen port (default 7860)
65
+ - `VEILRENDER_MAX_CONCURRENT` β€” Max concurrent pages (default 5)
66
+ - `VEILRENDER_TIMEOUT` β€” Page load timeout in ms (default 30000)
67
+
68
+ ### Worker pool
69
+ - `VEILRENDER_WORKERS` β€” Comma-separated remote CDP endpoints (enables pool mode)
70
+ - `VEILRENDER_WORKER_MAX_CONCURRENT` β€” Per-worker concurrency limit
71
+ - `VEILRENDER_WORKER_HEALTH_INTERVAL` β€” Health check interval in seconds (default 10)
72
+
73
+ ### Browser binary
74
+ - `CLOAKBROWSER_BINARY` β€” Path to custom browser binary
75
+ - `CLOAKBROWSER_MIRROR` β€” GitHub mirror for China downloads (e.g. `https://ghfast.top`)
76
+
77
  ## Files to never edit
78
 
79
  - `src/veilrender/_vendor/**` β€” vendored zerodep modules, update via
 
84
  1. `ruff check --fix && ruff format` on changed Python files
85
  2. Server starts and `/health` returns 200
86
  3. `/render` returns valid content for a test URL
87
+ 4. `/metrics` returns valid Prometheus output