serverless/README.md

ComfyUI serverless workers (RunPod)

Run ComfyUI workflows as RunPod serverless endpoints, optimized for cold start: everything is baked into the image (models included) so a worker whose host has the image cached starts in seconds and never touches the network.

Images: plx1029/comfyui-serverless on Docker Hub. Current endpoint:

endpoint	image tag	RunPod endpoint id	contents
qwen-edit-turbo	qwen-edit-turbo-v7	dom5lwr0o5wq6u	Qwen-Image-Edit-2511 bf16 + Lightning 4/8-step loras + Skin-Fix/Qwen4Play loras, workflow qwen-edit-turbo-v4
qwen-edit-nsfw	qwen-edit-nsfw-v1	tplcz9jtzihoxa	Qwen-Rapid-AIO-NSFW v11.4 + next-scene lora, RES4LYF sampler, workflow qwen-edit-nsfw
faceswap-lissie	faceswap-lissie-v1	qkeibkeoefcrk3	Flux-2-Klein 9B + Qwen3-8B TE + bfs-head/lissie loras + Qwen3-VL-4B + RMBG-2.0 (baked), workflow new-faceswap-lissie-v4

RunPod templates serverless-<endpoint>-v1, endpoints serverless-<endpoint>: H100 80GB (SXM/PCIe/NVL), FlashBoot, min 0 / max 2 workers, idle timeout 60 s, QUEUE_DELAY scaler. All endpoints are workflow-agnostic: they run their baked workflow by default, any other workflow whose models/nodes are in the image via "workflow", or a fully custom API-format graph via "workflow_json" (uploaded images are matched to LoadImage nodes by name).

Serverless-specific workflow tweaks (baked copies under endpoints/*/workflows/ differ from workflows/api/ in exactly these ways):

• qwen-edit-turbo (since -v7; -v1/-v2 had Amateur-Photo instead): lora slot 2 of the "User LoRAs" node is Qwen4Play_v2, off by default (params lora_qwen4play / lora_qwen4play_strength toggle it); the Amateur-Photo lora is not in the image anymore. Since -v7, start.sh passes no VRAM flag (ComfyUI default, same as pods): --highvram pinned lora weight backups in VRAM, and full-coverage loras like Qwen4Play (all 60 blocks) OOM an 80 GB H100 that way. Perf is unchanged (~56 s vs ~57 s cold execute).
• qwen-edit-nsfw: removed the KJNodes WidgetToString node (it reads UI-only metadata and crashes headless); the Image-Saver metadata node gets a static model-name string instead. Behavior is otherwise identical.
• faceswap-lissie: unchanged, but the endpoint defaults keep_vl_loaded=true so Qwen3-VL stays in VRAM between jobs (~10 s faster warm jobs).

Layout

serverless/
├── handler.py            # generic RunPod handler (same file for every endpoint)
├── start.sh              # container CMD: boot ComfyUI headless, run handler
├── build.sh              # daemonless image builder (crane; runs on a pod)
├── Dockerfile            # reproducible equivalent for machines with Docker
├── endpoints/
│   └── qwen-edit-turbo/
│       ├── endpoint.json           # models + custom nodes + default workflow
│       ├── workflows/<name>.json   # API-format workflow(s) baked into the image
│       └── params/<name>.json      # friendly-param -> node.input mapping
├── tests/
│   ├── local_test.py     # drive handler against a locally running ComfyUI
│   └── endpoint_test.py  # drive a deployed RunPod endpoint
└── client/               # standalone caller scripts (stdlib-only) + usage README
    ├── qwen_edit_sync.py     # /runsync: block until done
    ├── qwen_edit_async.py    # /run: submit, poll, fire-and-forget, collect, cancel
    └── qwen-edit-turbo-v4.json

Calling the endpoint

POST https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync     (or /run + /status/<id>)

{"input": {
  "images": [
    {"name": "main.jpg", "image": "<base64 or data-URI>"},
    {"name": "ref.png",  "image": "<base64>"}
  ],
  "params": {
    "prompt": "replace the outfit ...",
    "mode": "turbo-8",
    "seed": 123,
    "input_max_dim": 2048,
    "output_max_dim": 2560,
    "lora_skin_fix": true
  },
  "set": {"43.cfg": 1.0}
}}

• images[0] auto-fills the image slot, images[1] the image_ref slot; a missing second image automatically sets use_ref=false.
• mode: turbo-4 | turbo-8 (default) | quality.
• seed omitted → randomized (returned in the response).
• set patches any raw NODE.INPUT (or NODE.INPUT.KEY) after params.
• workflow selects among baked workflows; workflow_json runs a full API-format graph passthrough.
• Full param list per workflow: endpoints/<ep>/params/<workflow>.json.

Response: {"images": [{"filename", "type": "base64", "data"}], "seed", "prompt_id", "workflow", "timings": {...}}.

Payload limits (RunPod): ~10 MB on /run, ~20 MB on /runsync — enough for two photos. URL inputs / S3 outputs are a planned follow-up for bigger traffic.

Building images (no Docker needed — run on a pod)

cd serverless
# heavy tag: base + custom nodes + one layer PER model (build rarely)
./build.sh models qwen-edit-turbo v1
# tiny tag: + handler/workflows/params + CMD (build often, pushes only MBs)
./build.sh code qwen-edit-turbo v1 v1

Needs: crane + pigz installed, models on disk under $COMFY_WORKSPACE/ComfyUI/models/, DOCKER_HUB_ACCESS_TOKEN set. The model layers are content-addressed: rebuilding/pushing a new code version never re-uploads them. On a machine with Docker, Dockerfile builds the same image.

Adding a new endpoint family (e.g. krea2): create endpoints/krea2/{endpoint.json,workflows/,params/}, download its models on the build pod (download_missing_models.sh <workflows...>), then ./build.sh all krea2 v1. The handler is already generic.

Cold start design

• Models live in image layers → cached on RunPod host NVMe after first pull; no network volume, no downloads at start.
• ComfyUI runs headless (127.0.0.1, no Manager/frontend extras); its default memory management keeps the ~55 GB of bf16 weights resident in VRAM between jobs, so only the first job on a fresh worker pays model loading (no --highvram since v7 — see the lora note above).
• FlashBoot on the endpoint snapshots warm workers.
• SERVERLESS_REFRESH=true (endpoint env) re-pulls handler/workflows/params from hf.co/aleph65/ComfyUI/serverless/ at worker boot — iterate on code and workflow JSONs with zero image rebuild (off by default to keep cold starts fully offline).

Endpoint env vars

var	default	meaning
SERVERLESS_ENDPOINT	sole dir in endpoints/	which endpoint config to serve
COMFY_PORT	7865	internal ComfyUI port
COMFY_ARGS	(empty)	extra ComfyUI flags
COMFY_EXEC_TIMEOUT	600	per-job execution timeout (s)
SERVERLESS_REFRESH	false	boot-time code refresh from the HF repo