AGENTS.md

AGENTS.md — VeilRender

Context file for AI coding assistants. Symlinked as CLAUDE.md.

What this project is

VeilRender is a headless browser rendering API — accepts a URL, renders it with a stealth Chromium browser (CloakBrowser + Patchright), returns HTML/Markdown/readability content. Designed as a fallback for fetch tools that fail on JavaScript-rendered pages.

Supports two deployment modes: single-instance (embedded browser) and gateway + worker pool (browser containers scale independently).

Only runtime dependency is patchright. Everything else is vendored from zerodep — HTTP server, S3 client, HTML parsing, readability extraction.

Architecture

Request flow

HTTP request → auth → route handler → browser_manager.get_page()
                                        ↓
                              LocalWorker (embedded CloakBrowser)
                              RemoteWorker (CDP, Chromium)
                              PlaywrightWorker (Playwright protocol, Firefox/Camoufox)

Key components

Component	File	Purpose
HTTP server	_vendor/httpserver	Async HTTP framework (zerodep)
Browser pool	browser.py	LocalWorker, RemoteWorker, PlaywrightWorker, BrowserManager
Config	config.py	VEILRENDER_* env vars, worker protocol parsing
Routes	routes/*.py	/render, /screenshot, /health, /metrics, /stats, / dashboard
Storage	storage.py	L1 TTLCache + L2 S3 (vendored S3Client)
Filters	filters.py	Outbound request blocklist (StevenBlack/hosts)
CDP proxy	cdp_proxy.py	WebSocket proxy for direct CDP access
Stats	stats.py	In-memory counters, p50/p95 latency

Worker types

Type	Connection	Browser	Stealth
LocalWorker	Local subprocess	CloakBrowser (auto-downloaded)	Browser-level + driver-level
RemoteWorker	chromium.connect_over_cdp()	Any CDP-compatible Chromium	Driver-level (Patchright)
PlaywrightWorker	firefox.connect()	Camoufox or Playwright-served Firefox	Browser-level (Camoufox)

Repository layout

src/veilrender/
├── app.py              # HTTP server + CDP WebSocket multiplexing
├── config.py           # Settings from env vars (VEILRENDER_ prefix)
├── auth.py             # Token verification
├── browser.py          # Worker classes + BrowserManager pool coordinator
├── cdp_proxy.py        # CDP WebSocket proxy
├── filters.py          # Outbound request blocklist
├── models.py           # Request/response dataclasses
├── stats.py            # In-memory request counters and latency
├── storage.py          # L1 (TTLCache) + L2 (S3) render cache
├── routes/
│   ├── dashboard.py    # GET / — stats dashboard with i18n (en/zh)
│   ├── health.py       # GET /health
│   ├── metrics.py      # GET /metrics — Prometheus exposition
│   ├── render.py       # POST /render
│   └── screenshot.py   # POST /screenshot
└── _vendor/            # zerodep modules — DO NOT EDIT manually
    ├── httpserver.py    # Async HTTP server
    ├── cache.py         # TTLCache
    ├── s3.py            # S3-compatible client
    ├── readability.py   # Article extraction
    └── soup.py          # HTML parser
deploy/
├── compose.yaml            # Single-instance (prod)
├── compose-dev.yaml        # Dev instance (port 7861)
├── compose-pool.yaml       # Gateway + CloakBrowser workers
├── compose-pool-mixed.yaml # Gateway + CloakBrowser + Camoufox
└── Dockerfile.camoufox     # Camoufox server image
scripts/
└── download-cloakbrowser.py  # Download CloakBrowser binary

Commands

make dev            # Run dev server on :7860
make build          # Docker build (full target, with CloakBrowser)
make build-gateway  # Docker build (gateway target, no browser)
make run            # Docker run
make lint           # ruff check --fix && ruff format
make typecheck      # ty check
make vendor         # Re-vendor zerodep modules

Deployment

Dev deployment

make deploy-dev SSH_TARGET=oaklight.buttercup          # single-instance
make deploy-dev SSH_TARGET=oaklight.buttercup POOL=1   # gateway + workers

Builds from the working tree (not committed state), tags as dev-test, deploys to veilrender-dev stack (port 7861). Always verify no dirty files in src/ before deploying.

Release

1. Update CHANGELOG.md — move [Unreleased] to [X.Y.Z]
2. Bump __version__ in src/veilrender/__init__.py
3. Commit: git commit -m "bump version to X.Y.Z"
4. Tag and push: git tag vX.Y.Z && git push && git push origin vX.Y.Z
5. Release workflow auto-publishes to PyPI, Docker Hub, GHCR, and creates GitHub Release from changelog

Production

Prod runs on oaklight.buttercup at /dockervol/dockge/stacks/veilrender (port 7860), exposed via NPS as veilrender.service.oaklight.top.

Update prod to a released version:

ssh oaklight.buttercup 'cd /dockervol/dockge/stacks/veilrender && \
  docker compose pull && docker compose up -d'

Escalation

• Browser won't start → check _find_browser_binary() cascade: CLOAKBROWSER_BINARY env → ~/.cloakbrowser/*/chrome → auto-download
• Remote worker won't connect → check _resolve_to_ip() (Chromium rejects non-IP Host headers), verify CDP port is accessible
• Camoufox worker won't connect → check Playwright version compatibility (client/server major.minor must match)
• Vendor module issue → never fix in-place; update upstream zerodep, re-vendor
• Test failure after 3 attempts → stop, report full output
• Never: delete files to fix errors, skip lint, modify _vendor/ directly

Files to never edit

• src/veilrender/_vendor/** — vendored zerodep modules, update via make vendor or zerodep add from ~/projects/zerodep
• CLAUDE.md — symlink to AGENTS.md, edit AGENTS.md instead

Definition of done

1. ruff check --fix && ruff format on changed Python files
2. ty check passes
3. Server starts and /health returns 200
4. /render returns valid content for a test URL
5. /metrics returns valid Prometheus output