Spaces:
Paused
Paused
| # AGENTS.md — VeilRender | |
| > Context file for AI coding assistants. Symlinked as `CLAUDE.md`. | |
| ## What this project is | |
| VeilRender is a **headless browser rendering API** — accepts a URL, renders it | |
| with a stealth Chromium browser (CloakBrowser + Patchright), returns | |
| HTML/Markdown/readability content. Designed as a fallback for fetch tools that | |
| fail on JavaScript-rendered pages. | |
| Supports two deployment modes: **single-instance** (embedded browser) and | |
| **gateway + worker pool** (browser containers scale independently). | |
| Only runtime dependency is `patchright`. Everything else is vendored from | |
| [zerodep](https://github.com/Oaklight/zerodep) — HTTP server, S3 client, HTML | |
| parsing, readability extraction. | |
| ## Architecture | |
| ### Request flow | |
| ``` | |
| HTTP request → auth → route handler → browser_manager.get_page() | |
| ↓ | |
| LocalWorker (embedded CloakBrowser) | |
| RemoteWorker (CDP, Chromium) | |
| PlaywrightWorker (Playwright protocol, Firefox/Camoufox) | |
| ``` | |
| ### Key components | |
| | Component | File | Purpose | | |
| |-----------|------|---------| | |
| | HTTP server | `_vendor/httpserver` | Async HTTP framework (zerodep) | | |
| | Browser pool | `browser.py` | `LocalWorker`, `RemoteWorker`, `PlaywrightWorker`, `BrowserManager` | | |
| | Config | `config.py` | `VEILRENDER_*` env vars, worker protocol parsing | | |
| | Routes | `routes/*.py` | `/render`, `/screenshot`, `/health`, `/metrics`, `/stats`, `/` dashboard | | |
| | Storage | `storage.py` | L1 TTLCache + L2 S3 (vendored `S3Client`) | | |
| | Filters | `filters.py` | Outbound request blocklist (StevenBlack/hosts) | | |
| | CDP proxy | `cdp_proxy.py` | WebSocket proxy for direct CDP access | | |
| | Stats | `stats.py` | In-memory counters, p50/p95 latency | | |
| ### Worker types | |
| | Type | Connection | Browser | Stealth | | |
| |------|-----------|---------|---------| | |
| | `LocalWorker` | Local subprocess | CloakBrowser (auto-downloaded) | Browser-level + driver-level | | |
| | `RemoteWorker` | `chromium.connect_over_cdp()` | Any CDP-compatible Chromium | Driver-level (Patchright) | | |
| | `PlaywrightWorker` | `firefox.connect()` | Camoufox or Playwright-served Firefox | Browser-level (Camoufox) | | |
| ## Repository layout | |
| ``` | |
| src/veilrender/ | |
| ├── app.py # HTTP server + CDP WebSocket multiplexing | |
| ├── config.py # Settings from env vars (VEILRENDER_ prefix) | |
| ├── auth.py # Token verification | |
| ├── browser.py # Worker classes + BrowserManager pool coordinator | |
| ├── cdp_proxy.py # CDP WebSocket proxy | |
| ├── filters.py # Outbound request blocklist | |
| ├── models.py # Request/response dataclasses | |
| ├── stats.py # In-memory request counters and latency | |
| ├── storage.py # L1 (TTLCache) + L2 (S3) render cache | |
| ├── routes/ | |
| │ ├── dashboard.py # GET / — stats dashboard with i18n (en/zh) | |
| │ ├── health.py # GET /health | |
| │ ├── metrics.py # GET /metrics — Prometheus exposition | |
| │ ├── render.py # POST /render | |
| │ └── screenshot.py # POST /screenshot | |
| └── _vendor/ # zerodep modules — DO NOT EDIT manually | |
| ├── httpserver.py # Async HTTP server | |
| ├── cache.py # TTLCache | |
| ├── s3.py # S3-compatible client | |
| ├── readability.py # Article extraction | |
| └── soup.py # HTML parser | |
| deploy/ | |
| ├── compose.yaml # Single-instance (prod) | |
| ├── compose-dev.yaml # Dev instance (port 7861) | |
| ├── compose-pool.yaml # Gateway + CloakBrowser workers | |
| ├── compose-pool-mixed.yaml # Gateway + CloakBrowser + Camoufox | |
| └── Dockerfile.camoufox # Camoufox server image | |
| scripts/ | |
| └── download-cloakbrowser.py # Download CloakBrowser binary | |
| ``` | |
| ## Commands | |
| ```bash | |
| make dev # Run dev server on :7860 | |
| make build # Docker build (full target, with CloakBrowser) | |
| make build-gateway # Docker build (gateway target, no browser) | |
| make run # Docker run | |
| make lint # ruff check --fix && ruff format | |
| make typecheck # ty check | |
| make vendor # Re-vendor zerodep modules | |
| ``` | |
| ## Deployment | |
| ### Dev deployment | |
| ```bash | |
| make deploy-dev SSH_TARGET=oaklight.buttercup # single-instance | |
| make deploy-dev SSH_TARGET=oaklight.buttercup POOL=1 # gateway + workers | |
| ``` | |
| Builds from the **working tree** (not committed state), tags as `dev-test`, | |
| deploys to `veilrender-dev` stack (port 7861). Always verify no dirty files | |
| in `src/` before deploying. | |
| ### Release | |
| 1. Update `CHANGELOG.md` — move `[Unreleased]` to `[X.Y.Z]` | |
| 2. Bump `__version__` in `src/veilrender/__init__.py` | |
| 3. Commit: `git commit -m "bump version to X.Y.Z"` | |
| 4. Tag and push: `git tag vX.Y.Z && git push && git push origin vX.Y.Z` | |
| 5. Release workflow auto-publishes to PyPI, Docker Hub, GHCR, and creates | |
| GitHub Release from changelog | |
| ### Production | |
| Prod runs on `oaklight.buttercup` at `/dockervol/dockge/stacks/veilrender` | |
| (port 7860), exposed via NPS as `veilrender.service.oaklight.top`. | |
| Update prod to a released version: | |
| ```bash | |
| ssh oaklight.buttercup 'cd /dockervol/dockge/stacks/veilrender && \ | |
| docker compose pull && docker compose up -d' | |
| ``` | |
| ## Escalation | |
| - Browser won't start → check `_find_browser_binary()` cascade: | |
| `CLOAKBROWSER_BINARY` env → `~/.cloakbrowser/*/chrome` → auto-download | |
| - Remote worker won't connect → check `_resolve_to_ip()` (Chromium rejects | |
| non-IP Host headers), verify CDP port is accessible | |
| - Camoufox worker won't connect → check Playwright version compatibility | |
| (client/server major.minor must match) | |
| - Vendor module issue → never fix in-place; update upstream zerodep, re-vendor | |
| - Test failure after 3 attempts → stop, report full output | |
| - Never: delete files to fix errors, skip lint, modify `_vendor/` directly | |
| ## Files to never edit | |
| - `src/veilrender/_vendor/**` — vendored zerodep modules, update via | |
| `make vendor` or `zerodep add` from ~/projects/zerodep | |
| - `CLAUDE.md` — symlink to `AGENTS.md`, edit `AGENTS.md` instead | |
| ## Definition of done | |
| 1. `ruff check --fix && ruff format` on changed Python files | |
| 2. `ty check` passes | |
| 3. Server starts and `/health` returns 200 | |
| 4. `/render` returns valid content for a test URL | |
| 5. `/metrics` returns valid Prometheus output | |