---
license: mit
library_name: onnx
pipeline_tag: image-segmentation
tags:
- onnx
- matting
- portrait-matting
- background-removal
- virtual-background
- video-matting
- webrtc
- face-landmarks
- optical-flow
- mediapipe-alternative
---
# Longpipe: real-time portrait matting (official ONNX)
Official ONNX exports of the [Longpipe](https://longpipe.dev) models: real-time person matting for
video calls and live video (virtual backgrounds, background blur), plus the face-keypoint, face-mesh
and optical-flow heads that power touch-up, auto-reframe and temporal stability.
**Try it in your browser:** [longpipe.dev/demo](https://longpipe.dev/demo), or the [demo Space](https://huggingface.co/spaces/Longpipe/longpipe-demo).
Longpipe is primarily a browser SDK ([`npm install longpipe`](https://www.npmjs.com/package/longpipe),
[GitHub](https://github.com/sb2702/longpipe), [live demo](https://longpipe.dev/demo)) that runs these
networks as hand-written WebGPU/WebGL2 shaders. These ONNX files let you use the same trained weights
anywhere ONNX Runtime runs: Python, native apps, servers, mobile, or `onnxruntime-web`.
- **License:** MIT, code *and* weights. Training data provenance: [longpipe.dev/docs/licensing](https://longpipe.dev/docs/licensing).
- **Weights:** the SDK's current production weights (weights release `v0.0.5`, served at
`cdn.longpipe.dev/models/v/0.0.5/` and used by SDK 0.0.16), exported directly from the PyTorch training
checkpoints that produce those files. Every ONNX file is checked against PyTorch (max abs diff ≤ 7e-5,
most ≤ 1e-6); see `onnx/manifest.json`.
## Quality and speed
Average alpha error on the P3M-10K validation set (499 images): every Longpipe tier beats MediaPipe
on both MAE and IoU.
In the browser, the SDK's zero-copy GPU pipeline is faster than MediaPipe or BodyPix despite running
larger models (browser numbers; ONNX Runtime speed depends on your runtime and hardware):
## Tiers
Five tiers trade quality for cost. Each is a small "base" network (EfficientNet-Lite encoder +
decoder) wrapped in a high-resolution U-Net wrapper that refines edges at canvas resolution.
| Tier | Canvas (landscape, H×W) | Base res | `matting` size | Intended hardware |
|---|---|---|---|---|
| `xs` | 240×384 | 80×128 | 3.6 MB | lowest-end devices |
| `small` | 224×384 | 112×192 | 3.6 MB | Chromebooks, low-end laptops |
| `medium` | 320×512 | 160×256 | 13.0 MB | mid-range |
| `large` | 400×640 | 160×256 | 14.2 MB | mid-range laptops |
| `xl` | 768×1280 | 192×320 | 29.2 MB | Apple Silicon, discrete GPUs |
Every model has a fixed input shape. Each tier ships a `landscape` and a `portrait` (H and W swapped)
version. Resize your frame to the canvas size (bilinear) before inference; the network does not need
the exact aspect ratio.
## Files
```
onnx//matting_.onnx canvas RGB -> refined alpha (what the SDK renders)
onnx//matting_base_.onnx base-res RGB -> alpha (faster, softer edges)
onnx//face_.onnx canvas RGB -> 5 face-keypoint heatmaps
onnx//flow_.onnx two canvas frames -> backward optical flow
onnx/landmark_mesh.onnx 256x256 face crop -> 478 face-mesh points (shared by all tiers)
onnx/manifest.json shapes, sha256, parity vs PyTorch
```
**All inputs are RGB in [0, 1], NCHW float32, batch 1.** No mean/std normalization needed (the
landmark model's ImageNet normalization is built into the graph).
| Graph | Input | Output |
|---|---|---|
| `matting` | `rgb` [1,3,H,W] at canvas res | `alpha` [1,1,H,W], in [0,1] |
| `matting_base` | `rgb` [1,3,h,w] at base res | `alpha` [1,1,h,w], in [0,1] |
| `face` | `rgb` [1,3,H,W] at canvas res | `heatmaps` [1,5,h',w'], probabilities (sigmoid applied). Channels: left eye, right eye, nose, left mouth corner, right mouth corner. Grid is base/8 (`medium`, `large`, `xl`) or base/4 (`xs`, `small`) |
| `flow` | `frames` [1,6,H,W] at canvas res: **previous** frame in channels 0–2, **current** frame in 3–5 | `flow` [1,2,h',w'] **backward** flow (dx, dy): for each point on the current frame's grid, where it was in the previous frame, in base-resolution pixels. Grid is base/4 (base/2 on `xs`). Use it to warp the previous alpha onto the current frame |
| `landmark_mesh` | `rgb` [1,3,256,256] face crop | `landmarks` [1,478,2], (x, y) in crop coordinates, 0–1 |
## Quick start (Python)
```python
import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
path = hf_hub_download("Longpipe/longpipe", "onnx/large/matting_landscape.onnx")
sess = ort.InferenceSession(path)
H, W = sess.get_inputs()[0].shape[2:] # 400, 640
frame = Image.open("me.jpg").convert("RGB")
x = np.asarray(frame.resize((W, H), Image.BILINEAR), np.float32).transpose(2, 0, 1)[None] / 255
alpha = sess.run(None, {"rgb": x})[0][0, 0] # [H, W] in [0, 1]
# composite onto a background at the original resolution
a = np.asarray(Image.fromarray((alpha * 255).astype(np.uint8)).resize(frame.size, Image.BILINEAR)) / 255
bg = np.zeros_like(np.asarray(frame), dtype=np.float32); bg[..., 1] = 200 # green
out = np.asarray(frame) * a[..., None] + bg * (1 - a[..., None])
Image.fromarray(out.astype(np.uint8)).save("composited.png")
```
## Face crop for the landmark model
The 478-point mesh model expects the crop the SDK makes from the `face` heatmaps:
1. For each of the 5 heatmap channels, find the peak, then refine it with a soft-argmax over a 7×7
window (±3 cells) around the peak. Don't use the hard argmax alone; it jitters.
2. Take the bounding box of the 5 refined points (in heatmap cells). The crop is a square centered
on that box, with side = 2.4 × max(box width, box height). Map the center by `(cell + 0.5) / grid size`.
3. Skip the face if the weakest of the 5 peaks is below 0.15.
4. Resample that square to 256×256 (bilinear) and run `landmark_mesh`.
## Video: temporal stability (how the SDK uses `flow`)
The matting networks are static (one frame in, one alpha out). The SDK gets temporally stable mattes
from the flow head: each frame it predicts backward flow from the current frame into the previous one, warps
the previous *stabilized* alpha along that flow, and blends it with the fresh prediction through a
flow-gated stabilizer (gated on flow magnitude and divergence, to avoid ghosting at occlusion edges).
On lower tiers the matting network runs only every 2nd or 3rd frame and the held alpha is warped forward
in between. The face heatmaps get the same treatment. This blending logic lives in the SDK (TypeScript/WGSL),
not in these graphs. For a single-file temporal step, see the community port below.
## Notes
- Exported with ONNX opset 17 (`large`: opset 18). Tested with ONNX Runtime 1.23 (CPU).
- Each `face` / `flow` / `matting` file contains its own copy of the tier's encoder. The SDK shares
one encoder pass across all heads; if you need that, run the SDK.
- Community port: [imbcmdth/longpipe-onnx](https://github.com/imbcmdth/longpipe-onnx) converted the same
weights by reconstructing them from the SDK's packed GPU format (xs/small/large/xl), and includes a
single-graph temporal step with the flow stabilizer. Thanks! This repo is the official export, made
directly from the training checkpoints, covering all five tiers in both orientations.
## Links
- Website and docs: [longpipe.dev](https://longpipe.dev) · [architecture](https://longpipe.dev/docs/architecture)
- SDK: [github.com/sb2702/longpipe](https://github.com/sb2702/longpipe) · [npm](https://www.npmjs.com/package/longpipe)
- Built by the founder of [Vectorly](https://www.crunchbase.com/organization/vectorly) (effects SDK, acquired 2021). Also used for people-mode background removal in [katana.video/remove-background](https://katana.video/remove-background).