--- license: mit library_name: onnx pipeline_tag: image-segmentation tags: - onnx - matting - portrait-matting - background-removal - virtual-background - video-matting - webrtc - face-landmarks - optical-flow - mediapipe-alternative --- # Longpipe: real-time portrait matting (official ONNX) Official ONNX exports of the [Longpipe](https://longpipe.dev) models: real-time person matting for video calls and live video (virtual backgrounds, background blur), plus the face-keypoint, face-mesh and optical-flow heads that power touch-up, auto-reframe and temporal stability. **Try it in your browser:** [longpipe.dev/demo](https://longpipe.dev/demo), or the [demo Space](https://huggingface.co/spaces/Longpipe/longpipe-demo). Longpipe is primarily a browser SDK ([`npm install longpipe`](https://www.npmjs.com/package/longpipe), [GitHub](https://github.com/sb2702/longpipe), [live demo](https://longpipe.dev/demo)) that runs these networks as hand-written WebGPU/WebGL2 shaders. These ONNX files let you use the same trained weights anywhere ONNX Runtime runs: Python, native apps, servers, mobile, or `onnxruntime-web`. - **License:** MIT, code *and* weights. Training data provenance: [longpipe.dev/docs/licensing](https://longpipe.dev/docs/licensing). - **Weights:** the SDK's current production weights (weights release `v0.0.5`, served at `cdn.longpipe.dev/models/v/0.0.5/` and used by SDK 0.0.16), exported directly from the PyTorch training checkpoints that produce those files. Every ONNX file is checked against PyTorch (max abs diff ≤ 7e-5, most ≤ 1e-6); see `onnx/manifest.json`. ## Quality and speed Average alpha error on the P3M-10K validation set (499 images): every Longpipe tier beats MediaPipe on both MAE and IoU. Longpipe vs MediaPipe quality on P3M-10K In the browser, the SDK's zero-copy GPU pipeline is faster than MediaPipe or BodyPix despite running larger models (browser numbers; ONNX Runtime speed depends on your runtime and hardware): Longpipe vs MediaPipe and BodyPix speed ## Tiers Five tiers trade quality for cost. Each is a small "base" network (EfficientNet-Lite encoder + decoder) wrapped in a high-resolution U-Net wrapper that refines edges at canvas resolution. | Tier | Canvas (landscape, H×W) | Base res | `matting` size | Intended hardware | |---|---|---|---|---| | `xs` | 240×384 | 80×128 | 3.6 MB | lowest-end devices | | `small` | 224×384 | 112×192 | 3.6 MB | Chromebooks, low-end laptops | | `medium` | 320×512 | 160×256 | 13.0 MB | mid-range | | `large` | 400×640 | 160×256 | 14.2 MB | mid-range laptops | | `xl` | 768×1280 | 192×320 | 29.2 MB | Apple Silicon, discrete GPUs | Every model has a fixed input shape. Each tier ships a `landscape` and a `portrait` (H and W swapped) version. Resize your frame to the canvas size (bilinear) before inference; the network does not need the exact aspect ratio. ## Files ``` onnx//matting_.onnx canvas RGB -> refined alpha (what the SDK renders) onnx//matting_base_.onnx base-res RGB -> alpha (faster, softer edges) onnx//face_.onnx canvas RGB -> 5 face-keypoint heatmaps onnx//flow_.onnx two canvas frames -> backward optical flow onnx/landmark_mesh.onnx 256x256 face crop -> 478 face-mesh points (shared by all tiers) onnx/manifest.json shapes, sha256, parity vs PyTorch ``` **All inputs are RGB in [0, 1], NCHW float32, batch 1.** No mean/std normalization needed (the landmark model's ImageNet normalization is built into the graph). | Graph | Input | Output | |---|---|---| | `matting` | `rgb` [1,3,H,W] at canvas res | `alpha` [1,1,H,W], in [0,1] | | `matting_base` | `rgb` [1,3,h,w] at base res | `alpha` [1,1,h,w], in [0,1] | | `face` | `rgb` [1,3,H,W] at canvas res | `heatmaps` [1,5,h',w'], probabilities (sigmoid applied). Channels: left eye, right eye, nose, left mouth corner, right mouth corner. Grid is base/8 (`medium`, `large`, `xl`) or base/4 (`xs`, `small`) | | `flow` | `frames` [1,6,H,W] at canvas res: **previous** frame in channels 0–2, **current** frame in 3–5 | `flow` [1,2,h',w'] **backward** flow (dx, dy): for each point on the current frame's grid, where it was in the previous frame, in base-resolution pixels. Grid is base/4 (base/2 on `xs`). Use it to warp the previous alpha onto the current frame | | `landmark_mesh` | `rgb` [1,3,256,256] face crop | `landmarks` [1,478,2], (x, y) in crop coordinates, 0–1 | ## Quick start (Python) ```python import numpy as np, onnxruntime as ort from huggingface_hub import hf_hub_download from PIL import Image path = hf_hub_download("Longpipe/longpipe", "onnx/large/matting_landscape.onnx") sess = ort.InferenceSession(path) H, W = sess.get_inputs()[0].shape[2:] # 400, 640 frame = Image.open("me.jpg").convert("RGB") x = np.asarray(frame.resize((W, H), Image.BILINEAR), np.float32).transpose(2, 0, 1)[None] / 255 alpha = sess.run(None, {"rgb": x})[0][0, 0] # [H, W] in [0, 1] # composite onto a background at the original resolution a = np.asarray(Image.fromarray((alpha * 255).astype(np.uint8)).resize(frame.size, Image.BILINEAR)) / 255 bg = np.zeros_like(np.asarray(frame), dtype=np.float32); bg[..., 1] = 200 # green out = np.asarray(frame) * a[..., None] + bg * (1 - a[..., None]) Image.fromarray(out.astype(np.uint8)).save("composited.png") ``` ## Face crop for the landmark model The 478-point mesh model expects the crop the SDK makes from the `face` heatmaps: 1. For each of the 5 heatmap channels, find the peak, then refine it with a soft-argmax over a 7×7 window (±3 cells) around the peak. Don't use the hard argmax alone; it jitters. 2. Take the bounding box of the 5 refined points (in heatmap cells). The crop is a square centered on that box, with side = 2.4 × max(box width, box height). Map the center by `(cell + 0.5) / grid size`. 3. Skip the face if the weakest of the 5 peaks is below 0.15. 4. Resample that square to 256×256 (bilinear) and run `landmark_mesh`. ## Video: temporal stability (how the SDK uses `flow`) The matting networks are static (one frame in, one alpha out). The SDK gets temporally stable mattes from the flow head: each frame it predicts backward flow from the current frame into the previous one, warps the previous *stabilized* alpha along that flow, and blends it with the fresh prediction through a flow-gated stabilizer (gated on flow magnitude and divergence, to avoid ghosting at occlusion edges). On lower tiers the matting network runs only every 2nd or 3rd frame and the held alpha is warped forward in between. The face heatmaps get the same treatment. This blending logic lives in the SDK (TypeScript/WGSL), not in these graphs. For a single-file temporal step, see the community port below. ## Notes - Exported with ONNX opset 17 (`large`: opset 18). Tested with ONNX Runtime 1.23 (CPU). - Each `face` / `flow` / `matting` file contains its own copy of the tier's encoder. The SDK shares one encoder pass across all heads; if you need that, run the SDK. - Community port: [imbcmdth/longpipe-onnx](https://github.com/imbcmdth/longpipe-onnx) converted the same weights by reconstructing them from the SDK's packed GPU format (xs/small/large/xl), and includes a single-graph temporal step with the flow stabilizer. Thanks! This repo is the official export, made directly from the training checkpoints, covering all five tiers in both orientations. ## Links - Website and docs: [longpipe.dev](https://longpipe.dev) · [architecture](https://longpipe.dev/docs/architecture) - SDK: [github.com/sb2702/longpipe](https://github.com/sb2702/longpipe) · [npm](https://www.npmjs.com/package/longpipe) - Built by the founder of [Vectorly](https://www.crunchbase.com/organization/vectorly) (effects SDK, acquired 2021). Also used for people-mode background removal in [katana.video/remove-background](https://katana.video/remove-background).