|
Download README.md from Longpipe/longpipe: direct link, hf CLI and curl.
- Browser
- Download file 8.27 kB
-
https://huggingface.co/Longpipe/longpipe/resolve/main/README.md
- Command line
-
hf download hf://Longpipe/longpipe/README.md
-
curl -L -o README.md https://huggingface.co/Longpipe/longpipe/resolve/main/README.md
8.27 kB
| license: mit | |
| library_name: onnx | |
| pipeline_tag: image-segmentation | |
| tags: | |
| - onnx | |
| - matting | |
| - portrait-matting | |
| - background-removal | |
| - virtual-background | |
| - video-matting | |
| - webrtc | |
| - face-landmarks | |
| - optical-flow | |
| - mediapipe-alternative | |
| # Longpipe: real-time portrait matting (official ONNX) | |
| Official ONNX exports of the [Longpipe](https://longpipe.dev) models: real-time person matting for | |
| video calls and live video (virtual backgrounds, background blur), plus the face-keypoint, face-mesh | |
| and optical-flow heads that power touch-up, auto-reframe and temporal stability. | |
| **Try it in your browser:** [longpipe.dev/demo](https://longpipe.dev/demo), or the [demo Space](https://huggingface.co/spaces/Longpipe/longpipe-demo). | |
| Longpipe is primarily a browser SDK ([`npm install longpipe`](https://www.npmjs.com/package/longpipe), | |
| [GitHub](https://github.com/sb2702/longpipe), [live demo](https://longpipe.dev/demo)) that runs these | |
| networks as hand-written WebGPU/WebGL2 shaders. These ONNX files let you use the same trained weights | |
| anywhere ONNX Runtime runs: Python, native apps, servers, mobile, or `onnxruntime-web`. | |
| - **License:** MIT, code *and* weights. Training data provenance: [longpipe.dev/docs/licensing](https://longpipe.dev/docs/licensing). | |
| - **Weights:** the SDK's current production weights (weights release `v0.0.5`, served at | |
| `cdn.longpipe.dev/models/v/0.0.5/` and used by SDK 0.0.16), exported directly from the PyTorch training | |
| checkpoints that produce those files. Every ONNX file is checked against PyTorch (max abs diff ≤ 7e-5, | |
| most ≤ 1e-6); see `onnx/manifest.json`. | |
| ## Quality and speed | |
| Average alpha error on the P3M-10K validation set (499 images): every Longpipe tier beats MediaPipe | |
| on both MAE and IoU. | |
| <img width="640" alt="Longpipe vs MediaPipe quality on P3M-10K" src="https://github.com/user-attachments/assets/c7044937-95af-496f-893a-a05a50d7c914" /> | |
| In the browser, the SDK's zero-copy GPU pipeline is faster than MediaPipe or BodyPix despite running | |
| larger models (browser numbers; ONNX Runtime speed depends on your runtime and hardware): | |
| <img width="640" alt="Longpipe vs MediaPipe and BodyPix speed" src="https://github.com/user-attachments/assets/ce658b8d-dc62-4d6b-90f8-cd62dfef2a59" /> | |
| ## Tiers | |
| Five tiers trade quality for cost. Each is a small "base" network (EfficientNet-Lite encoder + | |
| decoder) wrapped in a high-resolution U-Net wrapper that refines edges at canvas resolution. | |
| | Tier | Canvas (landscape, H×W) | Base res | `matting` size | Intended hardware | | |
| |---|---|---|---|---| | |
| | `xs` | 240×384 | 80×128 | 3.6 MB | lowest-end devices | | |
| | `small` | 224×384 | 112×192 | 3.6 MB | Chromebooks, low-end laptops | | |
| | `medium` | 320×512 | 160×256 | 13.0 MB | mid-range | | |
| | `large` | 400×640 | 160×256 | 14.2 MB | mid-range laptops | | |
| | `xl` | 768×1280 | 192×320 | 29.2 MB | Apple Silicon, discrete GPUs | | |
| Every model has a fixed input shape. Each tier ships a `landscape` and a `portrait` (H and W swapped) | |
| version. Resize your frame to the canvas size (bilinear) before inference; the network does not need | |
| the exact aspect ratio. | |
| ## Files | |
| ``` | |
| onnx/<tier>/matting_<orientation>.onnx canvas RGB -> refined alpha (what the SDK renders) | |
| onnx/<tier>/matting_base_<orientation>.onnx base-res RGB -> alpha (faster, softer edges) | |
| onnx/<tier>/face_<orientation>.onnx canvas RGB -> 5 face-keypoint heatmaps | |
| onnx/<tier>/flow_<orientation>.onnx two canvas frames -> backward optical flow | |
| onnx/landmark_mesh.onnx 256x256 face crop -> 478 face-mesh points (shared by all tiers) | |
| onnx/manifest.json shapes, sha256, parity vs PyTorch | |
| ``` | |
| **All inputs are RGB in [0, 1], NCHW float32, batch 1.** No mean/std normalization needed (the | |
| landmark model's ImageNet normalization is built into the graph). | |
| | Graph | Input | Output | | |
| |---|---|---| | |
| | `matting` | `rgb` [1,3,H,W] at canvas res | `alpha` [1,1,H,W], in [0,1] | | |
| | `matting_base` | `rgb` [1,3,h,w] at base res | `alpha` [1,1,h,w], in [0,1] | | |
| | `face` | `rgb` [1,3,H,W] at canvas res | `heatmaps` [1,5,h',w'], probabilities (sigmoid applied). Channels: left eye, right eye, nose, left mouth corner, right mouth corner. Grid is base/8 (`medium`, `large`, `xl`) or base/4 (`xs`, `small`) | | |
| | `flow` | `frames` [1,6,H,W] at canvas res: **previous** frame in channels 0–2, **current** frame in 3–5 | `flow` [1,2,h',w'] **backward** flow (dx, dy): for each point on the current frame's grid, where it was in the previous frame, in base-resolution pixels. Grid is base/4 (base/2 on `xs`). Use it to warp the previous alpha onto the current frame | | |
| | `landmark_mesh` | `rgb` [1,3,256,256] face crop | `landmarks` [1,478,2], (x, y) in crop coordinates, 0–1 | | |
| ## Quick start (Python) | |
| ```python | |
| import numpy as np, onnxruntime as ort | |
| from huggingface_hub import hf_hub_download | |
| from PIL import Image | |
| path = hf_hub_download("Longpipe/longpipe", "onnx/large/matting_landscape.onnx") | |
| sess = ort.InferenceSession(path) | |
| H, W = sess.get_inputs()[0].shape[2:] # 400, 640 | |
| frame = Image.open("me.jpg").convert("RGB") | |
| x = np.asarray(frame.resize((W, H), Image.BILINEAR), np.float32).transpose(2, 0, 1)[None] / 255 | |
| alpha = sess.run(None, {"rgb": x})[0][0, 0] # [H, W] in [0, 1] | |
| # composite onto a background at the original resolution | |
| a = np.asarray(Image.fromarray((alpha * 255).astype(np.uint8)).resize(frame.size, Image.BILINEAR)) / 255 | |
| bg = np.zeros_like(np.asarray(frame), dtype=np.float32); bg[..., 1] = 200 # green | |
| out = np.asarray(frame) * a[..., None] + bg * (1 - a[..., None]) | |
| Image.fromarray(out.astype(np.uint8)).save("composited.png") | |
| ``` | |
| ## Face crop for the landmark model | |
| The 478-point mesh model expects the crop the SDK makes from the `face` heatmaps: | |
| 1. For each of the 5 heatmap channels, find the peak, then refine it with a soft-argmax over a 7×7 | |
| window (±3 cells) around the peak. Don't use the hard argmax alone; it jitters. | |
| 2. Take the bounding box of the 5 refined points (in heatmap cells). The crop is a square centered | |
| on that box, with side = 2.4 × max(box width, box height). Map the center by `(cell + 0.5) / grid size`. | |
| 3. Skip the face if the weakest of the 5 peaks is below 0.15. | |
| 4. Resample that square to 256×256 (bilinear) and run `landmark_mesh`. | |
| ## Video: temporal stability (how the SDK uses `flow`) | |
| The matting networks are static (one frame in, one alpha out). The SDK gets temporally stable mattes | |
| from the flow head: each frame it predicts backward flow from the current frame into the previous one, warps | |
| the previous *stabilized* alpha along that flow, and blends it with the fresh prediction through a | |
| flow-gated stabilizer (gated on flow magnitude and divergence, to avoid ghosting at occlusion edges). | |
| On lower tiers the matting network runs only every 2nd or 3rd frame and the held alpha is warped forward | |
| in between. The face heatmaps get the same treatment. This blending logic lives in the SDK (TypeScript/WGSL), | |
| not in these graphs. For a single-file temporal step, see the community port below. | |
| ## Notes | |
| - Exported with ONNX opset 17 (`large`: opset 18). Tested with ONNX Runtime 1.23 (CPU). | |
| - Each `face` / `flow` / `matting` file contains its own copy of the tier's encoder. The SDK shares | |
| one encoder pass across all heads; if you need that, run the SDK. | |
| - Community port: [imbcmdth/longpipe-onnx](https://github.com/imbcmdth/longpipe-onnx) converted the same | |
| weights by reconstructing them from the SDK's packed GPU format (xs/small/large/xl), and includes a | |
| single-graph temporal step with the flow stabilizer. Thanks! This repo is the official export, made | |
| directly from the training checkpoints, covering all five tiers in both orientations. | |
| ## Links | |
| - Website and docs: [longpipe.dev](https://longpipe.dev) · [architecture](https://longpipe.dev/docs/architecture) | |
| - SDK: [github.com/sb2702/longpipe](https://github.com/sb2702/longpipe) · [npm](https://www.npmjs.com/package/longpipe) | |
| - Built by the founder of [Vectorly](https://www.crunchbase.com/organization/vectorly) (effects SDK, acquired 2021). Also used for people-mode background removal in [katana.video/remove-background](https://katana.video/remove-background). | |