README: describe the output outside the refiner band precisely

75bb21d verified about 20 hours ago

11.6 kB

---
license: other
license_name: corridor-key-licence
license_link: LICENSE
library_name: pytorch
pipeline_tag: image-segmentation
tags:
  - matting
  - alpha-matting
  - green-screen
  - chroma-key
  - keying
  - real-time
  - obs
  - corridorkey
datasets:
  - mantelmoritz/CorridorKey-Synthetic-Streamers
---
# CorridorKey Real-Time Edge Refiner

**A 23k-parameter CNN that brings back native-resolution hair and edge detail when
[CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) runs at reduced resolution for live
green-screen keying, e.g. a 4K or 1080p facecam in OBS at 30–60 fps.**

![Input and real-time result on real 4K green-screen footage](images/hero_druid_4k.png)

<sub>Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time
pipeline: CorridorKey 2.0 Stage 1 on a 512×512 person crop + this refiner.</sub>

## What problem it solves

CorridorKey 2.0 produces excellent mattes, but at full 4K it is an offline model. To key live video, the
network has to run on a small grid: for example a 512×512 crop around the person, or 512×256 for the whole
frame. Its matte then has to be scaled back up to the camera resolution, and that step destroys the edge:

- **Bilinear upsampling** gives a blurry, blocky edge with a green or dark halo.
- **A colour guided filter** recovers some structure, but it smears hair, leaves fringes and fails where foreground
  and screen have similar brightness.
- **This refiner** looks at the native-resolution pixels and learns how CorridorKey 2.0 would have keyed them at full
  resolution. It predicts corrections for alpha *and* foreground colour (despill / unmixing) at every native pixel.

![1:1 native-pixel crops on real 4K footage](images/compare_druid_4k.png)

Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest
neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, the target the refiner was trained on.

## How it works

The refiner is a fully convolutional network: 11 input channels, a 3×3 input convolution, four residual dilated
3×3 blocks (dilations 1, 2, 4, 1), GELU, and a 1×1 output convolution. It has 24 channels in v10w and 16 in v9b.
Its receptive field is ±9 pixels. Per native pixel it receives:

| Channels | Input |
|---|---|
| 0–2 | `I`: native RGB frame (sRGB) |
| 3–5 | `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background |
| 6–8 | `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low − I_low)`) |
| 9 | `a_up`: CorridorKey alpha, bilinear upsampled |
| 10 | `a_gf`: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha |

It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is used only inside an
uncertainty band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of
CorridorKey's matte, which in solid foreground and clean background is CorridorKey's own 0 or 1. The band is weighted
by nearby foreground, which keeps the refiner out of dark or noisy screen areas.

In the real-time engine the refiner only runs on 32×32 tiles that intersect the band, with a fixed tile budget
(most uncertain tiles first). In fp16 that costs about 5–6 ms per frame with 256 tiles and 10–11 ms with 512 tiles
on an RTX 5070 Ti.
`corridorkey_refiner.py` in this repository is a standalone full-frame reference implementation of the same maths.
On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha.

## Files

| File | Description |
|---|---|
| `corridorkey_edge_refiner_v10w.safetensors` | **recommended**, 24 channels, 23,332 parameters |
| `corridorkey_edge_refiner_v9b.safetensors` | 16 channels, 10,948 parameters, slightly cheaper, slightly worse |
| `corridorkey_refiner.py` | model definition, loader, input construction and full-frame `refine()` |
| `LICENSE` | Corridor Key Licence, Version 2.0 |

## Usage

You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included
here) to produce the low-resolution matte. Then:

```python
import torch
from corridorkey_refiner import load_refiner, refine
net = load_refiner("corridorkey_edge_refiner_v10w.safetensors")   # fp16, CUDA
# rgb:       (1, 3, H, W)   native frame, sRGB in [0, 1]
# rgb_low:   (1, 3, h, w)   the (crop of the) frame as CorridorKey saw it
# alpha_low: (1, 1, h', w') CorridorKey alpha at its output resolution (e.g. Stage-1 aux head, 256x256)
# fg_low:    (1, 3, h', w') CorridorKey straight foreground, sRGB
alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low)           # native resolution
```

If you key a crop, pass the native-resolution crop as `rgb` and paste the result back. Apply despill and alpha
contrast after the refiner, as you would for CorridorKey's own output.

## Results

Edge error = mean absolute alpha error inside the edge band (×1000), measured against CorridorKey 2.0 offline at
native resolution; flicker = temporal error of the frame-to-frame alpha change in the edge band (×1000). Lower is
better. 32 consecutive frames per segment. None of these clips were used for training.

- **Real stock footage:** 6 Pexels / Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p), curly / wavy / long hair,
  headphones, headset, glasses.
- **Real 4K, hair** and **Real 4K, transparency:** the two clips of the CorridorKey 1.0 Quality Benchmark
  (2 segments each). The druid has curly hair, a beard and fur; the clown has tulle and water glasses.
- **Generated streamers:** the 10 test clips of series 3 of the dataset. They are deliberately hard (props in front
  of the person, partially covered screens, people leaving the frame).

**Person crop, 512×512** (CorridorKey 2.0 Stage 1 on a tracked square around the person):

| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame |
|---|---|---|---|---|---|---|
| Bilinear | 121.3 | 91.2 | 80.1 | 144.7 | 75.3 | — |
| Colour guided filter | 51.2 | 45.3 | 64.1 | 116.8 | 33.1 | 14.4 |
| Edge Refiner v9b | 32.5 | 32.7 | **61.9** | 104.6 | 22.0 | 24.4 |
| **Edge Refiner v10w** | **30.9** | **30.6** | 62.3 | **101.2** | **20.5** | 25.8 |

**Full frame, 512×256** (cheaper, for 60 fps):

| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame |
|---|---|---|---|---|---|---|
| Bilinear | 174.7 | 117.8 | 106.7 | 190.1 | 91.5 | — |
| Colour guided filter | 61.3 | 53.4 | 76.4 | 144.0 | 34.9 | 11.8 |
| Edge Refiner v9b | 41.7 | 46.3 | 73.6 | 135.9 | 26.4 | 16.5 |
| **Edge Refiner v10w** | **39.4** | **45.2** | **73.5** | **133.4** | **25.0** | 17.6 |

GPU time is end-to-end per frame, CorridorKey 2.0 Stage 1 included, on an RTX 5070 Ti. It is the median over the
6 real stock clips, measured without video I/O. The refiner itself costs about 5–6 ms with a budget of 256 tiles
and 10–11 ms with 512 tiles. The tile count, not the frame resolution, dominates the cost.

- Compared with the guided filter, the refiner (v10w) reduces the edge error by 36–40 % on the real stock clips and
  by 15–32 % on the 4K druid clip. Flicker drops by 28–38 %.
- On hair (druid) the gain is largest. On semi-transparent material (clown) it is small: there the result is limited
  by what the low-resolution network sees.

![Generated streamer clip from the test split, 1:1 crops](images/compare_generated_streamer.png)

## Training

- **Target:** CorridorKey 2.0 offline at native resolution (alpha and foreground). The refiner is a student of
  CorridorKey: it cannot be sharper than its teacher, and it inherits the teacher's mistakes.
- **Inputs during training:** exactly what the real-time engine produces at run time. Training patches were cut
  by running the real-time pipeline (512×512 person crop and 512×256 full frame, Stage 1 only) over every
  training clip and computing the 11 input channels, mostly from the edge band.
- **Data:**
  - 32 segments of real stock green-screen footage (Pexels, Mixkit)
  - the same frames re-composited from the teacher's foreground and alpha over procedural green/teal screens, also
    passed through a "bad camera" model (sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG, exposure errors)
  - 196 clips of the [CorridorKey Synthetic Streamers](https://huggingface.co/datasets/mantelmoritz/CorridorKey-Synthetic-Streamers)
    dataset, without real-room clips and without clips where the teacher keys green clothing away
  Real footage is sampled 3× more often than its share. 110,000 patches, 24,000 steps, loss on alpha, alpha
  gradients, composite and foreground.
- **Evaluation data was never used for training.**

## Limitations

![Limitation: free-flying strands far from the head](images/compare_facecam_1080.png)

- **The teacher is the ceiling.** Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too.
- **It can only refine what the low-resolution network found.** Single strands far outside CorridorKey's
  low-resolution matte are outside the refiner band and are lost, as in the image above. Semi-transparent objects
  (glass, tulle, water) depend almost entirely on the low-resolution network.
- Trained for **green (and teal / blue-green) screens** with people in front, mostly facecam and upper-body framing.
  It is not a general background-removal model.
- The numbers above compare against CorridorKey 2.0, not against hand-made ground truth.

## License

**Corridor Key Licence, Version 2.0** (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full
text is in [`LICENSE`](LICENSE).

This refiner was trained on the outputs of CorridorKey 2.0 and only works together with it. It is therefore
released as a CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4,
§5). In short:

- **Free of charge.** You may not sell it, repackage it as a standalone or competing product, or offer it as a paid
  API or inference service.
- **Share-alike:** derivatives must be released under the same licence, free of charge, and keep the
  "CorridorKey" name.
- Images and video you produce with it are yours.
- Commercial software integrations require a separate agreement with Corridor Digital
  (contact@corridordigital.com). Organisations above US $20M revenue need CorridorKey plugin seats
  (https://corridorkey.com).

### Attribution and modifications

- **CorridorKey 2.0** © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration
  with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0 ·
  https://corridorkey.com
- CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google
  DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository.
- **Changes relative to CorridorKey 2.0:** this repository adds a new, separately trained edge-refinement network and
  its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed.
- Example footage: CorridorKey 1.0 Quality Benchmark
  (https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels
  licence), and the CorridorKey Synthetic Streamers dataset.
- This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor
  Digital LLC.

	---
	license: other
	license_name: corridor-key-licence
	license_link: LICENSE
	library_name: pytorch
	pipeline_tag: image-segmentation
	tags:
	- matting
	- alpha-matting
	- green-screen
	- chroma-key
	- keying
	- real-time
	- obs
	- corridorkey
	datasets:
	- mantelmoritz/CorridorKey-Synthetic-Streamers
	---

	# CorridorKey Real-Time Edge Refiner

	**A 23k-parameter CNN that brings back native-resolution hair and edge detail when
	[CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) runs at reduced resolution for live
	green-screen keying, e.g. a 4K or 1080p facecam in OBS at 30–60 fps.**

	![Input and real-time result on real 4K green-screen footage](images/hero_druid_4k.png)

	<sub>Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time
	pipeline: CorridorKey 2.0 Stage 1 on a 512×512 person crop + this refiner.</sub>

	## What problem it solves

	CorridorKey 2.0 produces excellent mattes, but at full 4K it is an offline model. To key live video, the
	network has to run on a small grid: for example a 512×512 crop around the person, or 512×256 for the whole
	frame. Its matte then has to be scaled back up to the camera resolution, and that step destroys the edge:

	- Bilinear upsampling gives a blurry, blocky edge with a green or dark halo.
	- A colour guided filter recovers some structure, but it smears hair, leaves fringes and fails where foreground
	and screen have similar brightness.
	- This refiner looks at the native-resolution pixels and learns how CorridorKey 2.0 would have keyed them at full
	resolution. It predicts corrections for alpha and foreground colour (despill / unmixing) at every native pixel.

	![1:1 native-pixel crops on real 4K footage](images/compare_druid_4k.png)

	Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest
	neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, the target the refiner was trained on.

	## How it works

	The refiner is a fully convolutional network: 11 input channels, a 3×3 input convolution, four residual dilated
	3×3 blocks (dilations 1, 2, 4, 1), GELU, and a 1×1 output convolution. It has 24 channels in v10w and 16 in v9b.
	Its receptive field is ±9 pixels. Per native pixel it receives:

	\| Channels \| Input \|
	\|---\|---\|
	\| 0–2 \| `I`: native RGB frame (sRGB) \|
	\| 3–5 \| `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background \|
	\| 6–8 \| `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low − I_low)`) \|
	\| 9 \| `a_up`: CorridorKey alpha, bilinear upsampled \|
	\| 10 \| `a_gf`: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha \|

	It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is used only inside an
	uncertainty band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of
	CorridorKey's matte, which in solid foreground and clean background is CorridorKey's own 0 or 1. The band is weighted
	by nearby foreground, which keeps the refiner out of dark or noisy screen areas.

	In the real-time engine the refiner only runs on 32×32 tiles that intersect the band, with a fixed tile budget
	(most uncertain tiles first). In fp16 that costs about 5–6 ms per frame with 256 tiles and 10–11 ms with 512 tiles
	on an RTX 5070 Ti.
	`corridorkey_refiner.py` in this repository is a standalone full-frame reference implementation of the same maths.
	On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha.

	## Files

	\| File \| Description \|
	\|---\|---\|
	\| `corridorkey_edge_refiner_v10w.safetensors` \| recommended, 24 channels, 23,332 parameters \|
	\| `corridorkey_edge_refiner_v9b.safetensors` \| 16 channels, 10,948 parameters, slightly cheaper, slightly worse \|
	\| `corridorkey_refiner.py` \| model definition, loader, input construction and full-frame `refine()` \|
	\| `LICENSE` \| Corridor Key Licence, Version 2.0 \|

	## Usage

	You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included
	here) to produce the low-resolution matte. Then:

	```python
	import torch
	from corridorkey_refiner import load_refiner, refine

	net = load_refiner("corridorkey_edge_refiner_v10w.safetensors") # fp16, CUDA

	# rgb: (1, 3, H, W) native frame, sRGB in [0, 1]
	# rgb_low: (1, 3, h, w) the (crop of the) frame as CorridorKey saw it
	# alpha_low: (1, 1, h', w') CorridorKey alpha at its output resolution (e.g. Stage-1 aux head, 256x256)
	# fg_low: (1, 3, h', w') CorridorKey straight foreground, sRGB
	alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low) # native resolution
	```

	If you key a crop, pass the native-resolution crop as `rgb` and paste the result back. Apply despill and alpha
	contrast after the refiner, as you would for CorridorKey's own output.

	## Results

	Edge error = mean absolute alpha error inside the edge band (×1000), measured against CorridorKey 2.0 offline at
	native resolution; flicker = temporal error of the frame-to-frame alpha change in the edge band (×1000). Lower is
	better. 32 consecutive frames per segment. None of these clips were used for training.

	- Real stock footage: 6 Pexels / Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p), curly / wavy / long hair,
	headphones, headset, glasses.
	- Real 4K, hair and Real 4K, transparency: the two clips of the CorridorKey 1.0 Quality Benchmark
	(2 segments each). The druid has curly hair, a beard and fur; the clown has tulle and water glasses.
	- Generated streamers: the 10 test clips of series 3 of the dataset. They are deliberately hard (props in front
	of the person, partially covered screens, people leaving the frame).

	Person crop, 512×512 (CorridorKey 2.0 Stage 1 on a tracked square around the person):

	\| Upsampling \| Real stock \| Real 4K, hair \| Real 4K, transparency \| Generated streamers \| Flicker (real stock) \| GPU ms / frame \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Bilinear \| 121.3 \| 91.2 \| 80.1 \| 144.7 \| 75.3 \| — \|
	\| Colour guided filter \| 51.2 \| 45.3 \| 64.1 \| 116.8 \| 33.1 \| 14.4 \|
	\| Edge Refiner v9b \| 32.5 \| 32.7 \| 61.9 \| 104.6 \| 22.0 \| 24.4 \|
	\| Edge Refiner v10w \| 30.9 \| 30.6 \| 62.3 \| 101.2 \| 20.5 \| 25.8 \|

	Full frame, 512×256 (cheaper, for 60 fps):

	\| Upsampling \| Real stock \| Real 4K, hair \| Real 4K, transparency \| Generated streamers \| Flicker (real stock) \| GPU ms / frame \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Bilinear \| 174.7 \| 117.8 \| 106.7 \| 190.1 \| 91.5 \| — \|
	\| Colour guided filter \| 61.3 \| 53.4 \| 76.4 \| 144.0 \| 34.9 \| 11.8 \|
	\| Edge Refiner v9b \| 41.7 \| 46.3 \| 73.6 \| 135.9 \| 26.4 \| 16.5 \|
	\| Edge Refiner v10w \| 39.4 \| 45.2 \| 73.5 \| 133.4 \| 25.0 \| 17.6 \|

	GPU time is end-to-end per frame, CorridorKey 2.0 Stage 1 included, on an RTX 5070 Ti. It is the median over the
	6 real stock clips, measured without video I/O. The refiner itself costs about 5–6 ms with a budget of 256 tiles
	and 10–11 ms with 512 tiles. The tile count, not the frame resolution, dominates the cost.

	- Compared with the guided filter, the refiner (v10w) reduces the edge error by 36–40 % on the real stock clips and
	by 15–32 % on the 4K druid clip. Flicker drops by 28–38 %.
	- On hair (druid) the gain is largest. On semi-transparent material (clown) it is small: there the result is limited
	by what the low-resolution network sees.

	![Generated streamer clip from the test split, 1:1 crops](images/compare_generated_streamer.png)

	## Training

	- Target: CorridorKey 2.0 offline at native resolution (alpha and foreground). The refiner is a student of
	CorridorKey: it cannot be sharper than its teacher, and it inherits the teacher's mistakes.
	- Inputs during training: exactly what the real-time engine produces at run time. Training patches were cut
	by running the real-time pipeline (512×512 person crop and 512×256 full frame, Stage 1 only) over every
	training clip and computing the 11 input channels, mostly from the edge band.
	- Data:
	- 32 segments of real stock green-screen footage (Pexels, Mixkit)
	- the same frames re-composited from the teacher's foreground and alpha over procedural green/teal screens, also
	passed through a "bad camera" model (sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG, exposure errors)
	- 196 clips of the [CorridorKey Synthetic Streamers](https://huggingface.co/datasets/mantelmoritz/CorridorKey-Synthetic-Streamers)
	dataset, without real-room clips and without clips where the teacher keys green clothing away

	Real footage is sampled 3× more often than its share. 110,000 patches, 24,000 steps, loss on alpha, alpha
	gradients, composite and foreground.
	- Evaluation data was never used for training.

	## Limitations

	![Limitation: free-flying strands far from the head](images/compare_facecam_1080.png)

	- The teacher is the ceiling. Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too.
	- It can only refine what the low-resolution network found. Single strands far outside CorridorKey's
	low-resolution matte are outside the refiner band and are lost, as in the image above. Semi-transparent objects
	(glass, tulle, water) depend almost entirely on the low-resolution network.
	- Trained for green (and teal / blue-green) screens with people in front, mostly facecam and upper-body framing.
	It is not a general background-removal model.
	- The numbers above compare against CorridorKey 2.0, not against hand-made ground truth.

	## License

	Corridor Key Licence, Version 2.0 (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full
	text is in [`LICENSE`](LICENSE).

	This refiner was trained on the outputs of CorridorKey 2.0 and only works together with it. It is therefore
	released as a CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4,
	§5). In short:

	- Free of charge. You may not sell it, repackage it as a standalone or competing product, or offer it as a paid
	API or inference service.
	- Share-alike: derivatives must be released under the same licence, free of charge, and keep the
	"CorridorKey" name.
	- Images and video you produce with it are yours.
	- Commercial software integrations require a separate agreement with Corridor Digital
	(contact@corridordigital.com). Organisations above US $20M revenue need CorridorKey plugin seats
	(https://corridorkey.com).

	### Attribution and modifications

	- CorridorKey 2.0 © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration
	with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0 ·
	https://corridorkey.com
	- CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google
	DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository.
	- Changes relative to CorridorKey 2.0: this repository adds a new, separately trained edge-refinement network and
	its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed.
	- Example footage: CorridorKey 1.0 Quality Benchmark
	(https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels
	licence), and the CorridorKey Synthetic Streamers dataset.
	- This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor
	Digital LLC.