Revision 5: refiner graphs with Edge Refiner v15

121984e verified about 3 hours ago

16.5 kB

---
license: other
license_name: corridor-key-licence
license_link: LICENSE
library_name: onnx
pipeline_tag: image-segmentation
tags:
  - onnx
  - tensorrt
  - directml
  - matting
  - green-screen
  - keying
  - real-time
  - corridorkey
  - modnet
---
# CorridorKey ONNX Graphs

These are weight-free ONNX graphs of the real-time [CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0)
pipeline (MoGe-2 encoder, RVM step, Stage 1, Stage 2 and the
[CorridorKey Edge Refiner](https://huggingface.co/mantelmoritz/CorridorKey-Realtime-EdgeRefiner)), made for the
CorridorKey OBS plugin. They contain the network structure and small constants only. Every trained weight is
downloaded from its original publisher and bound locally, using the manifest that comes with each graph.

The repository also has graphs of CorridorKey 1.0 (green and blue, experimental) and of MODNet, which the plugin
uses to make CK1's coarse person hint. They are described in their [own section](#corridorkey-10-and-modnet).

![Facecam frame, PyTorch fp32 alpha, the same alpha from these graphs on TensorRT fp16, and the difference x 20](images/parity_trt_fp16.png)

<sub>Real outputs on one frame of a Pexels clip (Pexels licence), 256x512 grid. The right column is the absolute
difference multiplied by 20.</sub>

## Files

`graphs/<name>.onnx` is the graph and `graphs/<name>.json` its manifest. Two inference grids are included: 256x512
(the facecam profile, full frame) and 512x512 (person crop).

| Graph | Inputs | Outputs | Weights from |
|---|---|---|---|
| `moge_256x512`, `moge_512x512` | `rgb_moge` (1,3,H·7/32,W·7/32) | `moge_feat` (1,1,768,H/64,W/64) | MoGe-2 encoder, CK stats |
| `rvm_256x512`, `rvm_512x512` | `rgb_rvm` (1,3,H/4,W/4), `state` (1,1+H/64·W/64,384) | `rvm_feat` (1,1,384,H/64,W/64), `state_out` | RVM small, CK stats |
| `stage1nomask_256x512`, `stage1nomask_512x512` | `rgb` (1,3,H,W), `moge_feat`, `rvm_feat`, `hint` (1,1,1,H,W), `hint_quality` (1,1) | `baton` (1,1,512,H/16,W/16), `alpha` (1,1,H/2,W/2), `fg` (1,3,H/2,W/2) | CorridorKey 2.0 |
| `stage2nomask_256x512`, `stage2nomask_512x512` | `rgb` (1,3,H,W), `baton` | `alpha` (1,1,H,W), `fg` (1,3,H,W) | CorridorKey 2.0 |
| `stage1rvmnomask_256x512`, `stage1rvmnomask_512x512` (optional) | `rgb`, `state`, `moge_feat`, `hint`, `hint_quality` | `baton`, `alpha`, `fg`, `state_out` | CorridorKey 2.0, RVM, CK stats |
| `refiner_b256`, `_b384`, `_b512`, `_b768` | `x` (N,11,52,52) | `alpha` (N,1,52,52), `fg` (N,3,52,52) | Edge Refiner v15 |

All inputs and outputs are float32. `rgb`, `rgb_moge` and `rgb_rvm` are sRGB in [0, 1]. The MoGe and RVM inputs
are box-filter (area) downsamples of the `rgb` grid image. `fg` is straight (not premultiplied) sRGB in [0, 1];
the sRGB to linear to asinh conversion and its inverse happen inside the graphs. With no hint, `hint` is zeros and
`hint_quality` is -1. `state` starts as zeros and is carried from `state_out` to the next frame.

### The manifest

Each `.onnx` refers to its weights as standard ONNX external data in `<name>.weights`, which is not in this
repository. The manifest describes how to build that file:

- `inputs` and `outputs`: name, dtype, shape.
- `weights`: one entry per initializer with its name, dtype, shape, `offset` and `length` in `<name>.weights`
  (64-byte aligned), and `source` = `{file, key, ops}`.
- `sources`: the pinned downloads (repository or URL, revision, file name, size, sha256, licence).

`ops` turn the stored array into the initializer, in order: `["squeeze", axis]`, `["transpose", perm]`,
`["merge", a, b]` (merge axes [a, b) into one, C order) and `["zeros"]` (no source tensor). Only the RVM weights
need ops, because the original checkpoint is in JAX layout. The C++ binder of the OBS plugin and the Python
reference (`tools/onnx_weights.py`) produce byte-identical `.weights` files for all graphs.

### Weight sources

| id | File | From | Licence |
|---|---|---|---|
| `ck2`, `stats` | `corridorkey_v2.safetensors`, `foundation_stats.safetensors` | [nikopueringer/CorridorKey2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) | Corridor Key Licence |
| `moge` | `model.safetensors` | [mantelmoritz/MoGe-2-ViT-B-Encoder-safetensors](https://huggingface.co/mantelmoritz/MoGe-2-ViT-B-Encoder-safetensors) (format conversion of [Ruicheng/moge-2-vitb-normal](https://huggingface.co/Ruicheng/moge-2-vitb-normal)) | MIT (Microsoft), Apache-2.0 (DINOv2) |
| `rvm` | `pretrain_rvm_small16_256_204031069.npz` | [Google DeepMind representations4d](https://github.com/google-deepmind/representations4d) | CC BY 4.0 |
| `refiner` | `corridorkey_edge_refiner_v15.safetensors` | [mantelmoritz/CorridorKey-Realtime-EdgeRefiner](https://huggingface.co/mantelmoritz/CorridorKey-Realtime-EdgeRefiner) | Corridor Key Licence |
| `ck1green` | `CorridorKey_v1.0.safetensors` | [nikopueringer/CorridorKey_v1.0](https://huggingface.co/nikopueringer/CorridorKey_v1.0) | Corridor Key Licence |
| `ck1blue` | `CorridorKeyBlue_1.0.safetensors` | [nikopueringer/CorridorKeyBlue_1.0](https://huggingface.co/nikopueringer/CorridorKeyBlue_1.0) | Corridor Key Licence |
| `modnet` | `model.safetensors` | [mantelmoritz/MODNet-Webcam-safetensors](https://huggingface.co/mantelmoritz/MODNet-Webcam-safetensors) (format conversion of the official [MODNet](https://github.com/ZHKKKe/MODNet) webcam checkpoint) | Apache-2.0 |

The exact revisions and sha256 sums are in each manifest.

## How the graphs differ from upstream

The graphs were exported from the upstream PyTorch modules with `torch.onnx.export(dynamo=True)` at opset 18, in
float32. These changes are exact, apart from floating-point summation order:

- `stage1nomask` and `stage2nomask` run the shifted Swin windows without region masks, as the real-time engine
  does. Upstream applies the masks. The Edge Refiner was trained on the mask-free Stage-1 outputs, and with masks
  the real-time pipeline measured 3 to 6 % higher edge error against the offline teacher.
- The 7x7 depthwise SwiGLU convolutions run as two depthwise convolutions (value and gate) instead of one grouped
  convolution with two filters per channel. DirectML rejects the grouped form.
- The depatchify `ConvTranspose` (kernel 2s, stride s) is a 2x2 convolution, `DepthToSpace` and a crop. On
  TensorRT fp16 this makes Stage 2 about 15 % faster at the same accuracy. The idea comes from the export notes
  of [corridorkey-desktop](https://github.com/MarcelineVPQ/corridorkey-desktop); the code here is a separate
  implementation.
- Graph cleanup with the onnxscript optimizer and ONNX shape inference. The weights were graph inputs during this
  step, so no weight was folded into a constant.
- For DirectML: `Reshape` without `allowzero`, two-sided `Clip` as `Min`/`Max`, and rank-5 `MatMul` inputs
  flattened to rank 2 and reshaped back. A `Gemm` without a bias input (only in `modnet_*`) gets a scalar zero
  bias, because DirectML rejects the bias-free form once it fuses the `Gemm` with the following `Relu`.
- `alpha` and `fg` are decoded inside the graph, so a consumer needs no CorridorKey colour constants.

## Measured

RTX 5070 Ti, ONNX Runtime 1.31.1 with TensorRT 10 (TensorRT execution provider) and ONNX Runtime DirectML 1.24.4.
Device time per call with inputs and outputs bound on the GPU, median of 40 runs (DirectML: 10). Errors are
against the PyTorch fp32 modules on a real facecam frame. The desktop was in use during the runs, so expect about
±1 ms of noise.

| Graph (256x512) | TensorRT fp16 | TensorRT fp32 | DirectML fp32 | alpha error, TensorRT fp16 (max / mean) |
|---|---|---|---|---|
| `stage1nomask` | 7 to 8 ms | 10.4 ms | 15.6 ms | 8.0e-3 / 3.2e-4 |
| `stage2nomask` | 3.8 ms | 8.6 ms | 18.1 ms | 1.0e-2 / 2.4e-4 |
| `rvm` | 2.1 ms | 2.8 ms | 3.4 ms | (feature error 8.5e-3 of range in fp16) |
| `moge` | 2.0 ms | 2.6 ms | 3.3 ms | (feature error 4.5e-2 of range in fp16) |
| `refiner_b384` | 2.3 ms | 5.1 ms | 16.5 ms | 2.7e-3 / 2.8e-4 |

Recommended precisions: fp16 for Stage 1, Stage 2 and the refiner; fp32 for RVM and MoGe. In fp16 the MoGe features
drift by up to 4.5 % of their range, and RVM's error builds up through the recurrent state. bf16 was worse than
fp16 in every graph and does not build for Stage 1 and Stage 2. The CUDA execution provider without TensorRT
works (fp32 error about 1e-3) but is 4 to 8 times slower.

At 512x512: `stage1nomask` 8.0 ms, `stage2nomask` 6.5 ms on TensorRT fp16; 21.8 ms and 34.5 ms on DirectML.

### With CUDA graph capture

The TensorRT execution provider can capture each engine as a CUDA graph (`trt_cuda_graph_enable=1`). Inputs and
outputs then have to stay bound to the same GPU buffers. Most of the time above is kernel launch overhead, so
capture helps a lot, with identical outputs:

| Graph | Precision | Without capture | With capture |
|---|---|---|---|
| `rvm_256x512` | fp32 | 2.24 ms | 0.87 ms |
| `rvm_512x512` | fp32 | 3.38 ms | 0.89 ms |
| `stage1nomask_256x512` | fp16 | 7.29 ms | 3.81 ms |
| `stage1nomask_512x512` | fp16 | 10.42 ms | 4.73 ms |
| `stage2nomask_256x512` | fp16 | 4.13 ms | 3.39 ms |
| `refiner_b384` (v12) | fp16 | 1.87 ms | 1.80 ms |

`moge_*` does not run under capture yet; it is only needed every few frames.

## CorridorKey 1.0 and MODNet

![Facecam frame, the MODNet matte, the CK1 alpha keyed with that matte as hint, and the CK1 difference against PyTorch x 20](images/ck1_modnet_trt_fp16.png)

<sub>Real outputs on the same Pexels frame, squashed to the 512x512 grid and shown at 16:9 again. MODNet makes the
hint and CK1 keys with it, both on TensorRT fp16. The right column is the CK1 alpha difference against PyTorch
fp32, multiplied by 20.</sub>

| Graph | Inputs | Outputs | Weights from |
|---|---|---|---|
| `modnet_256`, `modnet_384`, `modnet_512` | `rgb` (1,3,S,S) | `matte` (1,1,S,S) | MODNet webcam |
| `ck1green_256`, `ck1green_384`, `ck1green_512` | `rgb` (1,3,S,S), `hint` (1,1,S,S) | `alpha` (1,1,S,S), `fg` (1,3,S,S) | CorridorKey 1.0 |
| `ck1blue_256`, `ck1blue_384`, `ck1blue_512` | same as `ck1green` | same as `ck1green` | CorridorKey Blue 1.0 |

S is the grid size in the name. Like CK1's own pipeline, the graphs take the whole frame squashed to a square, so
the alpha has to be scaled back to the frame size afterwards. `rgb` is sRGB in [0, 1] and the input normalisation
is in the graph, for MODNet as well. `hint` is a coarse person mask in [0, 1]. `fg` is straight sRGB in [0, 1].
CK1's loader resizes its position embedding (bicubic, from the 512x512 training grid) when the model is loaded.
Here that resize is part of the graph and runs on the weight itself, so TensorRT and ONNX Runtime fold it when the
session is created and the manifest keeps the weight in its original shape.

The plugin uses the MODNet matte as the CK1 hint because it needs no screen colour. A hint from a colour threshold
on the frame border was clearly worse in our tests (edge error 184 instead of 85 at 512).

CK1 is marked experimental. On our real facecam clips it has more edge error than the real-time CorridorKey 2.0
pipeline, even at its native 2048 grid. Edge error is the mean absolute alpha error inside the edge band, times
1000, against CorridorKey 2.0's offline output at full resolution. That reference favours CorridorKey 2.0, so
read the comparison with this in mind.

| Keyer | Edge error |
|---|---|
| CK1 green at 256, MODNet hint | 154 |
| CK1 green at 384, MODNet hint | 111 |
| CK1 green at 512, MODNet hint | 85 |
| CK1 green at its native 2048 (about 600 ms per frame in PyTorch), MODNet hint | 54 |
| CorridorKey 2.0 real-time, facecam profile | 34.8 |
| CorridorKey 2.0 real-time, balanced profile | 29.2 |

Speed and parity were measured as above, on the same frame squashed to each grid (milliseconds per call, without
CUDA graph capture):

| Graph | TensorRT fp16 | TensorRT fp32 | DirectML fp32 | max error, TensorRT fp16 |
|---|---|---|---|---|
| `modnet_256` | 1.8 ms | 2.8 ms | 1.9 ms | 4.0e-3 |
| `modnet_384` | 1.6 ms | 3.7 ms | 2.5 ms | 2.5e-3 |
| `modnet_512` | 2.1 ms | 4.2 ms | 3.2 ms | 2.9e-3 |
| `ck1green_256` | 4.7 ms | 6.5 ms | 8.9 ms | 2.2e-3 |
| `ck1green_384` | 5.4 ms | 9.7 ms | 15.8 ms | 4.9e-3 |
| `ck1green_512` | 7.1 ms | 16.8 ms | 28.5 ms | 2.7e-3 |
| `ck1blue_256` | 3.5 ms | 6.1 ms | 8.7 ms | 5.2e-3 |
| `ck1blue_384` | 4.5 ms | 9.6 ms | 16.0 ms | 4.7e-3 |
| `ck1blue_512` | 6.8 ms | 16.5 ms | 28.5 ms | 4.0e-3 |

The error column is the matte for MODNet and the alpha for CK1. The mean alpha error of CK1 on TensorRT fp16 is
about 1e-5, and its `fg` stays within 1e-2 of its range. On TensorRT fp32 and DirectML all graphs stay below
5e-3. All of them run entirely on DirectML.

## Changes

- Revision 5: `refiner_*` now use Edge Refiner v15 (32 channels instead of 24). The inputs and outputs are
  the same; the weights table and the pinned source in the manifests changed. If you cache TensorRT engines, key the
  cache on the weight file as well. ONNX Runtime's engine cache only hashes the graph, so a cached v12 engine would
  still be used after the update and give v12 results. The other graphs are unchanged.
- Revision 4: new `ck1green_*`, `ck1blue_*` and `modnet_*` graphs at 256, 384 and 512. The CorridorKey 2.0 graphs
  are unchanged.
- Revision 3: `moge_*` without the antialiased Resize to the size the graph is already fed at (DirectML ran that
  node on the CPU). `rvm_*` stacks the per-step output instead of writing it into an empty tensor, which exported
  as ScatterElements and also ran on the CPU under DirectML. Both graphs now run entirely on DirectML, with
  unchanged outputs and weights. New optional `stage1rvmnomask_*`: the RVM step and Stage 1 in one graph, with
  the RVM input computed in the graph as the 4x4 average of `rgb`. It saves one call per frame, but a weakly typed
  TensorRT fp16 build also runs RVM in fp16, so the separate graphs stay the default.
- Revision 2: `stage1nomask` and `stage2nomask` carry their RoPE tables in float32. Revision 1 had bfloat16 tables
  by mistake. Parity against the PyTorch fp32 reference stays in the same range (TensorRT fp32 alpha max 8e-4,
  fp16 8e-3). The manifests and the weights are unchanged.
- Revision 1: first release.

## Known limits

- The graphs need a fixed grid. Other grid sizes need a new export.
- DirectML runs everything, but Stage 1 and Stage 2 are 2 to 5 times slower than TensorRT on the same card. AMD
  and MIGraphX were not tested.
- The parity numbers come from one frame per grid. The end-to-end comparison on the evaluation clips runs in the
  PyTorch engine of the research repository.

## Licence and attribution

The graphs are derived from CorridorKey 2.0 and published under the Corridor Key Licence, Version 2.0
(CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full text is in [`LICENSE`](LICENSE).
Derivatives must stay free of charge, use the same licence and keep the CorridorKey name. Commercial software
integrations need a separate agreement with Corridor Digital (contact@corridordigital.com).

- CorridorKey 2.0 © Corridor Digital LLC, designed and trained by Zachary Harris in collaboration with Niko
  Pueringer and Corridor Digital: https://huggingface.co/nikopueringer/CorridorKey2.0, https://corridorkey.com
- MoGe-2 (Microsoft, MIT) with a DINOv2 backbone (Meta, Apache-2.0): https://github.com/microsoft/MoGe
- RVM (Google DeepMind; code Apache-2.0, weights CC BY 4.0): https://github.com/google-deepmind/representations4d
- CorridorKey 1.0 and CorridorKey Blue 1.0 © Corridor Digital LLC, Corridor Key Licence:
  https://huggingface.co/nikopueringer/CorridorKey_v1.0, https://huggingface.co/nikopueringer/CorridorKeyBlue_1.0
- MODNet by Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan and Rynson W.H. Lau, Apache-2.0:
  https://github.com/ZHKKKe/MODNet (commit 28165a45). The `modnet_*` graphs are derived from its code and are
  published under the Apache License 2.0, not the Corridor Key Licence. The licence text is in
  [`LICENSE-MODNet`](LICENSE-MODNet) and the notice in [`NOTICE-MODNet`](NOTICE-MODNet). Changes: the
  inference path of the webcam model was exported to ONNX with the input normalisation inside the graph, and the
  two bias-free `Gemm` nodes of its SE block got a zero bias for DirectML.
- No weights of any of these models are contained in this repository.
- This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor
  Digital LLC, Microsoft, Meta, Google DeepMind or the MODNet authors.

	---
	license: other
	license_name: corridor-key-licence
	license_link: LICENSE
	library_name: onnx
	pipeline_tag: image-segmentation
	tags:
	- onnx
	- tensorrt
	- directml
	- matting
	- green-screen
	- keying
	- real-time
	- corridorkey
	- modnet
	---

	# CorridorKey ONNX Graphs

	These are weight-free ONNX graphs of the real-time [CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0)
	pipeline (MoGe-2 encoder, RVM step, Stage 1, Stage 2 and the
	[CorridorKey Edge Refiner](https://huggingface.co/mantelmoritz/CorridorKey-Realtime-EdgeRefiner)), made for the
	CorridorKey OBS plugin. They contain the network structure and small constants only. Every trained weight is
	downloaded from its original publisher and bound locally, using the manifest that comes with each graph.

	The repository also has graphs of CorridorKey 1.0 (green and blue, experimental) and of MODNet, which the plugin
	uses to make CK1's coarse person hint. They are described in their [own section](#corridorkey-10-and-modnet).

	![Facecam frame, PyTorch fp32 alpha, the same alpha from these graphs on TensorRT fp16, and the difference x 20](images/parity_trt_fp16.png)

	<sub>Real outputs on one frame of a Pexels clip (Pexels licence), 256x512 grid. The right column is the absolute
	difference multiplied by 20.</sub>

	## Files

	`graphs/<name>.onnx` is the graph and `graphs/<name>.json` its manifest. Two inference grids are included: 256x512
	(the facecam profile, full frame) and 512x512 (person crop).

	\| Graph \| Inputs \| Outputs \| Weights from \|
	\|---\|---\|---\|---\|
	\| `moge_256x512`, `moge_512x512` \| `rgb_moge` (1,3,H·7/32,W·7/32) \| `moge_feat` (1,1,768,H/64,W/64) \| MoGe-2 encoder, CK stats \|
	\| `rvm_256x512`, `rvm_512x512` \| `rgb_rvm` (1,3,H/4,W/4), `state` (1,1+H/64·W/64,384) \| `rvm_feat` (1,1,384,H/64,W/64), `state_out` \| RVM small, CK stats \|
	\| `stage1nomask_256x512`, `stage1nomask_512x512` \| `rgb` (1,3,H,W), `moge_feat`, `rvm_feat`, `hint` (1,1,1,H,W), `hint_quality` (1,1) \| `baton` (1,1,512,H/16,W/16), `alpha` (1,1,H/2,W/2), `fg` (1,3,H/2,W/2) \| CorridorKey 2.0 \|
	\| `stage2nomask_256x512`, `stage2nomask_512x512` \| `rgb` (1,3,H,W), `baton` \| `alpha` (1,1,H,W), `fg` (1,3,H,W) \| CorridorKey 2.0 \|
	\| `stage1rvmnomask_256x512`, `stage1rvmnomask_512x512` (optional) \| `rgb`, `state`, `moge_feat`, `hint`, `hint_quality` \| `baton`, `alpha`, `fg`, `state_out` \| CorridorKey 2.0, RVM, CK stats \|
	\| `refiner_b256`, `_b384`, `_b512`, `_b768` \| `x` (N,11,52,52) \| `alpha` (N,1,52,52), `fg` (N,3,52,52) \| Edge Refiner v15 \|

	All inputs and outputs are float32. `rgb`, `rgb_moge` and `rgb_rvm` are sRGB in [0, 1]. The MoGe and RVM inputs
	are box-filter (area) downsamples of the `rgb` grid image. `fg` is straight (not premultiplied) sRGB in [0, 1];
	the sRGB to linear to asinh conversion and its inverse happen inside the graphs. With no hint, `hint` is zeros and
	`hint_quality` is -1. `state` starts as zeros and is carried from `state_out` to the next frame.

	### The manifest

	Each `.onnx` refers to its weights as standard ONNX external data in `<name>.weights`, which is not in this
	repository. The manifest describes how to build that file:

	- `inputs` and `outputs`: name, dtype, shape.
	- `weights`: one entry per initializer with its name, dtype, shape, `offset` and `length` in `<name>.weights`
	(64-byte aligned), and `source` = `{file, key, ops}`.
	- `sources`: the pinned downloads (repository or URL, revision, file name, size, sha256, licence).

	`ops` turn the stored array into the initializer, in order: `["squeeze", axis]`, `["transpose", perm]`,
	`["merge", a, b]` (merge axes [a, b) into one, C order) and `["zeros"]` (no source tensor). Only the RVM weights
	need ops, because the original checkpoint is in JAX layout. The C++ binder of the OBS plugin and the Python
	reference (`tools/onnx_weights.py`) produce byte-identical `.weights` files for all graphs.

	### Weight sources

	\| id \| File \| From \| Licence \|
	\|---\|---\|---\|---\|
	\| `ck2`, `stats` \| `corridorkey_v2.safetensors`, `foundation_stats.safetensors` \| [nikopueringer/CorridorKey2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) \| Corridor Key Licence \|
	\| `moge` \| `model.safetensors` \| [mantelmoritz/MoGe-2-ViT-B-Encoder-safetensors](https://huggingface.co/mantelmoritz/MoGe-2-ViT-B-Encoder-safetensors) (format conversion of [Ruicheng/moge-2-vitb-normal](https://huggingface.co/Ruicheng/moge-2-vitb-normal)) \| MIT (Microsoft), Apache-2.0 (DINOv2) \|
	\| `rvm` \| `pretrain_rvm_small16_256_204031069.npz` \| [Google DeepMind representations4d](https://github.com/google-deepmind/representations4d) \| CC BY 4.0 \|
	\| `refiner` \| `corridorkey_edge_refiner_v15.safetensors` \| [mantelmoritz/CorridorKey-Realtime-EdgeRefiner](https://huggingface.co/mantelmoritz/CorridorKey-Realtime-EdgeRefiner) \| Corridor Key Licence \|
	\| `ck1green` \| `CorridorKey_v1.0.safetensors` \| [nikopueringer/CorridorKey_v1.0](https://huggingface.co/nikopueringer/CorridorKey_v1.0) \| Corridor Key Licence \|
	\| `ck1blue` \| `CorridorKeyBlue_1.0.safetensors` \| [nikopueringer/CorridorKeyBlue_1.0](https://huggingface.co/nikopueringer/CorridorKeyBlue_1.0) \| Corridor Key Licence \|
	\| `modnet` \| `model.safetensors` \| [mantelmoritz/MODNet-Webcam-safetensors](https://huggingface.co/mantelmoritz/MODNet-Webcam-safetensors) (format conversion of the official [MODNet](https://github.com/ZHKKKe/MODNet) webcam checkpoint) \| Apache-2.0 \|

	The exact revisions and sha256 sums are in each manifest.

	## How the graphs differ from upstream

	The graphs were exported from the upstream PyTorch modules with `torch.onnx.export(dynamo=True)` at opset 18, in
	float32. These changes are exact, apart from floating-point summation order:

	- `stage1nomask` and `stage2nomask` run the shifted Swin windows without region masks, as the real-time engine
	does. Upstream applies the masks. The Edge Refiner was trained on the mask-free Stage-1 outputs, and with masks
	the real-time pipeline measured 3 to 6 % higher edge error against the offline teacher.
	- The 7x7 depthwise SwiGLU convolutions run as two depthwise convolutions (value and gate) instead of one grouped
	convolution with two filters per channel. DirectML rejects the grouped form.
	- The depatchify `ConvTranspose` (kernel 2s, stride s) is a 2x2 convolution, `DepthToSpace` and a crop. On
	TensorRT fp16 this makes Stage 2 about 15 % faster at the same accuracy. The idea comes from the export notes
	of [corridorkey-desktop](https://github.com/MarcelineVPQ/corridorkey-desktop); the code here is a separate
	implementation.
	- Graph cleanup with the onnxscript optimizer and ONNX shape inference. The weights were graph inputs during this
	step, so no weight was folded into a constant.
	- For DirectML: `Reshape` without `allowzero`, two-sided `Clip` as `Min`/`Max`, and rank-5 `MatMul` inputs
	flattened to rank 2 and reshaped back. A `Gemm` without a bias input (only in `modnet_*`) gets a scalar zero
	bias, because DirectML rejects the bias-free form once it fuses the `Gemm` with the following `Relu`.
	- `alpha` and `fg` are decoded inside the graph, so a consumer needs no CorridorKey colour constants.

	## Measured

	RTX 5070 Ti, ONNX Runtime 1.31.1 with TensorRT 10 (TensorRT execution provider) and ONNX Runtime DirectML 1.24.4.
	Device time per call with inputs and outputs bound on the GPU, median of 40 runs (DirectML: 10). Errors are
	against the PyTorch fp32 modules on a real facecam frame. The desktop was in use during the runs, so expect about
	±1 ms of noise.

	\| Graph (256x512) \| TensorRT fp16 \| TensorRT fp32 \| DirectML fp32 \| alpha error, TensorRT fp16 (max / mean) \|
	\|---\|---\|---\|---\|---\|
	\| `stage1nomask` \| 7 to 8 ms \| 10.4 ms \| 15.6 ms \| 8.0e-3 / 3.2e-4 \|
	\| `stage2nomask` \| 3.8 ms \| 8.6 ms \| 18.1 ms \| 1.0e-2 / 2.4e-4 \|
	\| `rvm` \| 2.1 ms \| 2.8 ms \| 3.4 ms \| (feature error 8.5e-3 of range in fp16) \|
	\| `moge` \| 2.0 ms \| 2.6 ms \| 3.3 ms \| (feature error 4.5e-2 of range in fp16) \|
	\| `refiner_b384` \| 2.3 ms \| 5.1 ms \| 16.5 ms \| 2.7e-3 / 2.8e-4 \|

	Recommended precisions: fp16 for Stage 1, Stage 2 and the refiner; fp32 for RVM and MoGe. In fp16 the MoGe features
	drift by up to 4.5 % of their range, and RVM's error builds up through the recurrent state. bf16 was worse than
	fp16 in every graph and does not build for Stage 1 and Stage 2. The CUDA execution provider without TensorRT
	works (fp32 error about 1e-3) but is 4 to 8 times slower.

	At 512x512: `stage1nomask` 8.0 ms, `stage2nomask` 6.5 ms on TensorRT fp16; 21.8 ms and 34.5 ms on DirectML.

	### With CUDA graph capture

	The TensorRT execution provider can capture each engine as a CUDA graph (`trt_cuda_graph_enable=1`). Inputs and
	outputs then have to stay bound to the same GPU buffers. Most of the time above is kernel launch overhead, so
	capture helps a lot, with identical outputs:

	\| Graph \| Precision \| Without capture \| With capture \|
	\|---\|---\|---\|---\|
	\| `rvm_256x512` \| fp32 \| 2.24 ms \| 0.87 ms \|
	\| `rvm_512x512` \| fp32 \| 3.38 ms \| 0.89 ms \|
	\| `stage1nomask_256x512` \| fp16 \| 7.29 ms \| 3.81 ms \|
	\| `stage1nomask_512x512` \| fp16 \| 10.42 ms \| 4.73 ms \|
	\| `stage2nomask_256x512` \| fp16 \| 4.13 ms \| 3.39 ms \|
	\| `refiner_b384` (v12) \| fp16 \| 1.87 ms \| 1.80 ms \|

	`moge_*` does not run under capture yet; it is only needed every few frames.

	## CorridorKey 1.0 and MODNet

	![Facecam frame, the MODNet matte, the CK1 alpha keyed with that matte as hint, and the CK1 difference against PyTorch x 20](images/ck1_modnet_trt_fp16.png)

	<sub>Real outputs on the same Pexels frame, squashed to the 512x512 grid and shown at 16:9 again. MODNet makes the
	hint and CK1 keys with it, both on TensorRT fp16. The right column is the CK1 alpha difference against PyTorch
	fp32, multiplied by 20.</sub>

	\| Graph \| Inputs \| Outputs \| Weights from \|
	\|---\|---\|---\|---\|
	\| `modnet_256`, `modnet_384`, `modnet_512` \| `rgb` (1,3,S,S) \| `matte` (1,1,S,S) \| MODNet webcam \|
	\| `ck1green_256`, `ck1green_384`, `ck1green_512` \| `rgb` (1,3,S,S), `hint` (1,1,S,S) \| `alpha` (1,1,S,S), `fg` (1,3,S,S) \| CorridorKey 1.0 \|
	\| `ck1blue_256`, `ck1blue_384`, `ck1blue_512` \| same as `ck1green` \| same as `ck1green` \| CorridorKey Blue 1.0 \|

	S is the grid size in the name. Like CK1's own pipeline, the graphs take the whole frame squashed to a square, so
	the alpha has to be scaled back to the frame size afterwards. `rgb` is sRGB in [0, 1] and the input normalisation
	is in the graph, for MODNet as well. `hint` is a coarse person mask in [0, 1]. `fg` is straight sRGB in [0, 1].
	CK1's loader resizes its position embedding (bicubic, from the 512x512 training grid) when the model is loaded.
	Here that resize is part of the graph and runs on the weight itself, so TensorRT and ONNX Runtime fold it when the
	session is created and the manifest keeps the weight in its original shape.

	The plugin uses the MODNet matte as the CK1 hint because it needs no screen colour. A hint from a colour threshold
	on the frame border was clearly worse in our tests (edge error 184 instead of 85 at 512).

	CK1 is marked experimental. On our real facecam clips it has more edge error than the real-time CorridorKey 2.0
	pipeline, even at its native 2048 grid. Edge error is the mean absolute alpha error inside the edge band, times
	1000, against CorridorKey 2.0's offline output at full resolution. That reference favours CorridorKey 2.0, so
	read the comparison with this in mind.

	\| Keyer \| Edge error \|
	\|---\|---\|
	\| CK1 green at 256, MODNet hint \| 154 \|
	\| CK1 green at 384, MODNet hint \| 111 \|
	\| CK1 green at 512, MODNet hint \| 85 \|
	\| CK1 green at its native 2048 (about 600 ms per frame in PyTorch), MODNet hint \| 54 \|
	\| CorridorKey 2.0 real-time, facecam profile \| 34.8 \|
	\| CorridorKey 2.0 real-time, balanced profile \| 29.2 \|

	Speed and parity were measured as above, on the same frame squashed to each grid (milliseconds per call, without
	CUDA graph capture):

	\| Graph \| TensorRT fp16 \| TensorRT fp32 \| DirectML fp32 \| max error, TensorRT fp16 \|
	\|---\|---\|---\|---\|---\|
	\| `modnet_256` \| 1.8 ms \| 2.8 ms \| 1.9 ms \| 4.0e-3 \|
	\| `modnet_384` \| 1.6 ms \| 3.7 ms \| 2.5 ms \| 2.5e-3 \|
	\| `modnet_512` \| 2.1 ms \| 4.2 ms \| 3.2 ms \| 2.9e-3 \|
	\| `ck1green_256` \| 4.7 ms \| 6.5 ms \| 8.9 ms \| 2.2e-3 \|
	\| `ck1green_384` \| 5.4 ms \| 9.7 ms \| 15.8 ms \| 4.9e-3 \|
	\| `ck1green_512` \| 7.1 ms \| 16.8 ms \| 28.5 ms \| 2.7e-3 \|
	\| `ck1blue_256` \| 3.5 ms \| 6.1 ms \| 8.7 ms \| 5.2e-3 \|
	\| `ck1blue_384` \| 4.5 ms \| 9.6 ms \| 16.0 ms \| 4.7e-3 \|
	\| `ck1blue_512` \| 6.8 ms \| 16.5 ms \| 28.5 ms \| 4.0e-3 \|

	The error column is the matte for MODNet and the alpha for CK1. The mean alpha error of CK1 on TensorRT fp16 is
	about 1e-5, and its `fg` stays within 1e-2 of its range. On TensorRT fp32 and DirectML all graphs stay below
	5e-3. All of them run entirely on DirectML.

	## Changes

	- Revision 5: `refiner_*` now use Edge Refiner v15 (32 channels instead of 24). The inputs and outputs are
	the same; the weights table and the pinned source in the manifests changed. If you cache TensorRT engines, key the
	cache on the weight file as well. ONNX Runtime's engine cache only hashes the graph, so a cached v12 engine would
	still be used after the update and give v12 results. The other graphs are unchanged.
	- Revision 4: new `ck1green_`, `ck1blue_` and `modnet_*` graphs at 256, 384 and 512. The CorridorKey 2.0 graphs
	are unchanged.
	- Revision 3: `moge_*` without the antialiased Resize to the size the graph is already fed at (DirectML ran that
	node on the CPU). `rvm_*` stacks the per-step output instead of writing it into an empty tensor, which exported
	as ScatterElements and also ran on the CPU under DirectML. Both graphs now run entirely on DirectML, with
	unchanged outputs and weights. New optional `stage1rvmnomask_*`: the RVM step and Stage 1 in one graph, with
	the RVM input computed in the graph as the 4x4 average of `rgb`. It saves one call per frame, but a weakly typed
	TensorRT fp16 build also runs RVM in fp16, so the separate graphs stay the default.
	- Revision 2: `stage1nomask` and `stage2nomask` carry their RoPE tables in float32. Revision 1 had bfloat16 tables
	by mistake. Parity against the PyTorch fp32 reference stays in the same range (TensorRT fp32 alpha max 8e-4,
	fp16 8e-3). The manifests and the weights are unchanged.
	- Revision 1: first release.

	## Known limits

	- The graphs need a fixed grid. Other grid sizes need a new export.
	- DirectML runs everything, but Stage 1 and Stage 2 are 2 to 5 times slower than TensorRT on the same card. AMD
	and MIGraphX were not tested.
	- The parity numbers come from one frame per grid. The end-to-end comparison on the evaluation clips runs in the
	PyTorch engine of the research repository.

	## Licence and attribution

	The graphs are derived from CorridorKey 2.0 and published under the Corridor Key Licence, Version 2.0
	(CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full text is in [`LICENSE`](LICENSE).
	Derivatives must stay free of charge, use the same licence and keep the CorridorKey name. Commercial software
	integrations need a separate agreement with Corridor Digital (contact@corridordigital.com).

	- CorridorKey 2.0 © Corridor Digital LLC, designed and trained by Zachary Harris in collaboration with Niko
	Pueringer and Corridor Digital: https://huggingface.co/nikopueringer/CorridorKey2.0, https://corridorkey.com
	- MoGe-2 (Microsoft, MIT) with a DINOv2 backbone (Meta, Apache-2.0): https://github.com/microsoft/MoGe
	- RVM (Google DeepMind; code Apache-2.0, weights CC BY 4.0): https://github.com/google-deepmind/representations4d
	- CorridorKey 1.0 and CorridorKey Blue 1.0 © Corridor Digital LLC, Corridor Key Licence:
	https://huggingface.co/nikopueringer/CorridorKey_v1.0, https://huggingface.co/nikopueringer/CorridorKeyBlue_1.0
	- MODNet by Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan and Rynson W.H. Lau, Apache-2.0:
	https://github.com/ZHKKKe/MODNet (commit 28165a45). The `modnet_*` graphs are derived from its code and are
	published under the Apache License 2.0, not the Corridor Key Licence. The licence text is in
	[`LICENSE-MODNet`](LICENSE-MODNet) and the notice in [`NOTICE-MODNet`](NOTICE-MODNet). Changes: the
	inference path of the webcam model was exported to ONNX with the input normalisation inside the graph, and the
	two bias-free `Gemm` nodes of its SE block got a zero bias for DirectML.
	- No weights of any of these models are contained in this repository.
	- This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor
	Digital LLC, Microsoft, Meta, Google DeepMind or the MODNet authors.