serverless/README.md

# ComfyUI serverless workers (RunPod)

Run ComfyUI workflows as RunPod serverless endpoints, optimized for cold start:
**everything is baked into the image** (models included) so a worker whose host
has the image cached starts in seconds and never touches the network.

Images: `plx1029/comfyui-serverless` on Docker Hub. Current endpoint:

| endpoint | image tag | RunPod endpoint id | contents |
|---|---|---|---|
| qwen-edit-turbo | `qwen-edit-turbo-v7` | `dom5lwr0o5wq6u` | Qwen-Image-Edit-2511 **bf16** + Lightning 4/8-step loras + Skin-Fix/Qwen4Play loras, workflow `qwen-edit-turbo-v4` |
| qwen-edit-nsfw | `qwen-edit-nsfw-v1` | `tplcz9jtzihoxa` | Qwen-Rapid-AIO-NSFW v11.4 + next-scene lora, RES4LYF sampler, workflow `qwen-edit-nsfw` |
| faceswap-lissie | `faceswap-lissie-v1` | `qkeibkeoefcrk3` | Flux-2-Klein 9B + Qwen3-8B TE + bfs-head/lissie loras + Qwen3-VL-4B + RMBG-2.0 (baked), workflow `new-faceswap-lissie-v4` |

RunPod templates `serverless-<endpoint>-v1`, endpoints `serverless-<endpoint>`:
H100 80GB (SXM/PCIe/NVL), FlashBoot, min 0 / max 2 workers, idle timeout 60 s,
QUEUE_DELAY scaler. All endpoints are **workflow-agnostic**: they run their
baked workflow by default, any other workflow whose models/nodes are in the
image via `"workflow"`, or a fully custom API-format graph via
`"workflow_json"` (uploaded images are matched to `LoadImage` nodes by name).

Serverless-specific workflow tweaks (baked copies under `endpoints/*/workflows/`
differ from `workflows/api/` in exactly these ways):
- `qwen-edit-turbo` (since `-v7`; `-v1`/`-v2` had Amateur-Photo instead): lora
  slot 2 of the "User LoRAs" node is `Qwen4Play_v2`, **off by default**
  (params `lora_qwen4play` / `lora_qwen4play_strength` toggle it); the
  Amateur-Photo lora is not in the image anymore. Since `-v7`, `start.sh`
  passes no VRAM flag (ComfyUI default, same as pods): `--highvram` pinned
  lora weight backups in VRAM, and full-coverage loras like Qwen4Play (all
  60 blocks) OOM an 80 GB H100 that way. Perf is unchanged (~56 s vs ~57 s
  cold execute).
- `qwen-edit-nsfw`: removed the KJNodes `WidgetToString` node (it reads
  UI-only metadata and crashes headless); the Image-Saver metadata node gets a
  static model-name string instead. Behavior is otherwise identical.
- `faceswap-lissie`: unchanged, but the endpoint defaults `keep_vl_loaded=true`
  so Qwen3-VL stays in VRAM between jobs (~10 s faster warm jobs).

## Layout

```
serverless/
├── handler.py            # generic RunPod handler (same file for every endpoint)
├── start.sh              # container CMD: boot ComfyUI headless, run handler
├── build.sh              # daemonless image builder (crane; runs on a pod)
├── Dockerfile            # reproducible equivalent for machines with Docker
├── endpoints/
│   └── qwen-edit-turbo/
│       ├── endpoint.json           # models + custom nodes + default workflow
│       ├── workflows/<name>.json   # API-format workflow(s) baked into the image
│       └── params/<name>.json      # friendly-param -> node.input mapping
├── tests/
│   ├── local_test.py     # drive handler against a locally running ComfyUI
│   └── endpoint_test.py  # drive a deployed RunPod endpoint
└── client/               # standalone caller scripts (stdlib-only) + usage README
    ├── qwen_edit_sync.py     # /runsync: block until done
    ├── qwen_edit_async.py    # /run: submit, poll, fire-and-forget, collect, cancel
    └── qwen-edit-turbo-v4.json
```

## Calling the endpoint

```
POST https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync     (or /run + /status/<id>)
```

```json
{"input": {
  "images": [
    {"name": "main.jpg", "image": "<base64 or data-URI>"},
    {"name": "ref.png",  "image": "<base64>"}
  ],
  "params": {
    "prompt": "replace the outfit ...",
    "mode": "turbo-8",
    "seed": 123,
    "input_max_dim": 2048,
    "output_max_dim": 2560,
    "lora_skin_fix": true
  },
  "set": {"43.cfg": 1.0}
}}
```

- `images[0]` auto-fills the `image` slot, `images[1]` the `image_ref` slot; a
  missing second image automatically sets `use_ref=false`.
- `mode`: `turbo-4` | `turbo-8` (default) | `quality`.
- `seed` omitted → randomized (returned in the response).
- `set` patches any raw `NODE.INPUT` (or `NODE.INPUT.KEY`) after params.
- `workflow` selects among baked workflows; `workflow_json` runs a full
  API-format graph passthrough.
- Full param list per workflow: `endpoints/<ep>/params/<workflow>.json`.

Response: `{"images": [{"filename", "type": "base64", "data"}], "seed",
"prompt_id", "workflow", "timings": {...}}`.

Payload limits (RunPod): ~10 MB on `/run`, ~20 MB on `/runsync` — enough for
two photos. URL inputs / S3 outputs are a planned follow-up for bigger traffic.

## Building images (no Docker needed — run on a pod)

```bash
cd serverless
# heavy tag: base + custom nodes + one layer PER model (build rarely)
./build.sh models qwen-edit-turbo v1
# tiny tag: + handler/workflows/params + CMD (build often, pushes only MBs)
./build.sh code qwen-edit-turbo v1 v1
```

Needs: `crane` + `pigz` installed, models on disk under
`$COMFY_WORKSPACE/ComfyUI/models/`, `DOCKER_HUB_ACCESS_TOKEN` set. The model
layers are content-addressed: rebuilding/pushing a new code version never
re-uploads them. On a machine with Docker, `Dockerfile` builds the same image.

**Adding a new endpoint family** (e.g. krea2): create
`endpoints/krea2/{endpoint.json,workflows/,params/}`, download its models on
the build pod (`download_missing_models.sh <workflows...>`), then
`./build.sh all krea2 v1`. The handler is already generic.

## Cold start design

- Models live in image layers → cached on RunPod host NVMe after first pull;
  no network volume, no downloads at start.
- ComfyUI runs headless (`127.0.0.1`, no Manager/frontend extras); its
  default memory management keeps the ~55 GB of bf16 weights resident in
  VRAM between jobs, so only the first job on a fresh worker pays model
  loading (no `--highvram` since v7 — see the lora note above).
- FlashBoot on the endpoint snapshots warm workers.
- `SERVERLESS_REFRESH=true` (endpoint env) re-pulls handler/workflows/params
  from `hf.co/aleph65/ComfyUI/serverless/` at worker boot — iterate on code and
  workflow JSONs with **zero image rebuild** (off by default to keep cold
  starts fully offline).

## Endpoint env vars

| var | default | meaning |
|---|---|---|
| `SERVERLESS_ENDPOINT` | sole dir in `endpoints/` | which endpoint config to serve |
| `COMFY_PORT` | 7865 | internal ComfyUI port |
| `COMFY_ARGS` | (empty) | extra ComfyUI flags |
| `COMFY_EXEC_TIMEOUT` | 600 | per-job execution timeout (s) |
| `SERVERLESS_REFRESH` | false | boot-time code refresh from the HF repo |