license: other
license_name: corridor-key-licence
license_link: LICENSE
library_name: pytorch
pipeline_tag: image-segmentation
tags:
- matting
- alpha-matting
- green-screen
- chroma-key
- keying
- real-time
- obs
- corridorkey
datasets:
- mantelmoritz/CorridorKey-Synthetic-Streamers
CorridorKey Real-Time Edge Refiner
A 23k-parameter CNN that brings back native-resolution hair and edge detail when CorridorKey 2.0 runs at reduced resolution for live green-screen keying, e.g. a 4K or 1080p facecam in OBS at 30–60 fps.
Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time pipeline: CorridorKey 2.0 Stage 1 on a 512×512 person crop + this refiner.
What problem it solves
CorridorKey 2.0 produces excellent mattes, but at full 4K it is an offline model. To key live video, the network has to run on a small grid: for example a 512×512 crop around the person, or 512×256 for the whole frame. Its matte then has to be scaled back up to the camera resolution, and that step destroys the edge:
- Bilinear upsampling gives a blurry, blocky edge with a green or dark halo.
- A colour guided filter recovers some structure, but it smears hair, leaves fringes and fails where foreground and screen have similar brightness.
- This refiner looks at the native-resolution pixels and learns how CorridorKey 2.0 would have keyed them at full resolution. It predicts corrections for alpha and foreground colour (despill / unmixing) at every native pixel.
Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, the target the refiner was trained on.
How it works
The refiner is a fully convolutional network: 11 input channels, a 3×3 input convolution, four residual dilated 3×3 blocks (dilations 1, 2, 4, 1), GELU, and a 1×1 output convolution. It has 24 channels in v10w and 16 in v9b. Its receptive field is ±9 pixels. Per native pixel it receives:
| Channels | Input |
|---|---|
| 0–2 | I: native RGB frame (sRGB) |
| 3–5 | B: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background |
| 6–8 | F: CorridorKey foreground transferred onto the native frame (I + upsample(F_low − I_low)) |
| 9 | a_up: CorridorKey alpha, bilinear upsampled |
| 10 | a_gf: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha |
It outputs residuals on top of a_gf (alpha) and F (foreground). The result is used only inside an
uncertainty band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of
CorridorKey's matte, which in solid foreground and clean background is CorridorKey's own 0 or 1. The band is weighted
by nearby foreground, which keeps the refiner out of dark or noisy screen areas.
In the real-time engine the refiner only runs on 32×32 tiles that intersect the band, with a fixed tile budget
(most uncertain tiles first). In fp16 that costs about 5–6 ms per frame with 256 tiles and 10–11 ms with 512 tiles
on an RTX 5070 Ti.
corridorkey_refiner.py in this repository is a standalone full-frame reference implementation of the same maths.
On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha.
Files
| File | Description |
|---|---|
corridorkey_edge_refiner_v10w.safetensors |
recommended, 24 channels, 23,332 parameters |
corridorkey_edge_refiner_v9b.safetensors |
16 channels, 10,948 parameters, slightly cheaper, slightly worse |
corridorkey_refiner.py |
model definition, loader, input construction and full-frame refine() |
LICENSE |
Corridor Key Licence, Version 2.0 |
Usage
You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included here) to produce the low-resolution matte. Then:
import torch
from corridorkey_refiner import load_refiner, refine
net = load_refiner("corridorkey_edge_refiner_v10w.safetensors") # fp16, CUDA
# rgb: (1, 3, H, W) native frame, sRGB in [0, 1]
# rgb_low: (1, 3, h, w) the (crop of the) frame as CorridorKey saw it
# alpha_low: (1, 1, h', w') CorridorKey alpha at its output resolution (e.g. Stage-1 aux head, 256x256)
# fg_low: (1, 3, h', w') CorridorKey straight foreground, sRGB
alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low) # native resolution
If you key a crop, pass the native-resolution crop as rgb and paste the result back. Apply despill and alpha
contrast after the refiner, as you would for CorridorKey's own output.
Results
Edge error = mean absolute alpha error inside the edge band (×1000), measured against CorridorKey 2.0 offline at native resolution; flicker = temporal error of the frame-to-frame alpha change in the edge band (×1000). Lower is better. 32 consecutive frames per segment. None of these clips were used for training.
- Real stock footage: 6 Pexels / Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p), curly / wavy / long hair, headphones, headset, glasses.
- Real 4K, hair and Real 4K, transparency: the two clips of the CorridorKey 1.0 Quality Benchmark (2 segments each). The druid has curly hair, a beard and fur; the clown has tulle and water glasses.
- Generated streamers: the 10 test clips of series 3 of the dataset. They are deliberately hard (props in front of the person, partially covered screens, people leaving the frame).
Person crop, 512×512 (CorridorKey 2.0 Stage 1 on a tracked square around the person):
| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame |
|---|---|---|---|---|---|---|
| Bilinear | 121.3 | 91.2 | 80.1 | 144.7 | 75.3 | — |
| Colour guided filter | 51.2 | 45.3 | 64.1 | 116.8 | 33.1 | 14.4 |
| Edge Refiner v9b | 32.5 | 32.7 | 61.9 | 104.6 | 22.0 | 24.4 |
| Edge Refiner v10w | 30.9 | 30.6 | 62.3 | 101.2 | 20.5 | 25.8 |
Full frame, 512×256 (cheaper, for 60 fps):
| Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame |
|---|---|---|---|---|---|---|
| Bilinear | 174.7 | 117.8 | 106.7 | 190.1 | 91.5 | — |
| Colour guided filter | 61.3 | 53.4 | 76.4 | 144.0 | 34.9 | 11.8 |
| Edge Refiner v9b | 41.7 | 46.3 | 73.6 | 135.9 | 26.4 | 16.5 |
| Edge Refiner v10w | 39.4 | 45.2 | 73.5 | 133.4 | 25.0 | 17.6 |
GPU time is end-to-end per frame, CorridorKey 2.0 Stage 1 included, on an RTX 5070 Ti. It is the median over the 6 real stock clips, measured without video I/O. The refiner itself costs about 5–6 ms with a budget of 256 tiles and 10–11 ms with 512 tiles. The tile count, not the frame resolution, dominates the cost.
- Compared with the guided filter, the refiner (v10w) reduces the edge error by 36–40 % on the real stock clips and by 15–32 % on the 4K druid clip. Flicker drops by 28–38 %.
- On hair (druid) the gain is largest. On semi-transparent material (clown) it is small: there the result is limited by what the low-resolution network sees.
Training
Target: CorridorKey 2.0 offline at native resolution (alpha and foreground). The refiner is a student of CorridorKey: it cannot be sharper than its teacher, and it inherits the teacher's mistakes.
Inputs during training: exactly what the real-time engine produces at run time. Training patches were cut by running the real-time pipeline (512×512 person crop and 512×256 full frame, Stage 1 only) over every training clip and computing the 11 input channels, mostly from the edge band.
Data:
- 32 segments of real stock green-screen footage (Pexels, Mixkit)
- the same frames re-composited from the teacher's foreground and alpha over procedural green/teal screens, also passed through a "bad camera" model (sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG, exposure errors)
- 196 clips of the CorridorKey Synthetic Streamers dataset, without real-room clips and without clips where the teacher keys green clothing away
Real footage is sampled 3× more often than its share. 110,000 patches, 24,000 steps, loss on alpha, alpha gradients, composite and foreground.
Evaluation data was never used for training.
Limitations
- The teacher is the ceiling. Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too.
- It can only refine what the low-resolution network found. Single strands far outside CorridorKey's low-resolution matte are outside the refiner band and are lost, as in the image above. Semi-transparent objects (glass, tulle, water) depend almost entirely on the low-resolution network.
- Trained for green (and teal / blue-green) screens with people in front, mostly facecam and upper-body framing. It is not a general background-removal model.
- The numbers above compare against CorridorKey 2.0, not against hand-made ground truth.
License
Corridor Key Licence, Version 2.0 (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full
text is in LICENSE.
This refiner was trained on the outputs of CorridorKey 2.0 and only works together with it. It is therefore released as a CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4, §5). In short:
- Free of charge. You may not sell it, repackage it as a standalone or competing product, or offer it as a paid API or inference service.
- Share-alike: derivatives must be released under the same licence, free of charge, and keep the "CorridorKey" name.
- Images and video you produce with it are yours.
- Commercial software integrations require a separate agreement with Corridor Digital (contact@corridordigital.com). Organisations above US $20M revenue need CorridorKey plugin seats (https://corridorkey.com).
Attribution and modifications
- CorridorKey 2.0 © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0 · https://corridorkey.com
- CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository.
- Changes relative to CorridorKey 2.0: this repository adds a new, separately trained edge-refinement network and its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed.
- Example footage: CorridorKey 1.0 Quality Benchmark (https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels licence), and the CorridorKey Synthetic Streamers dataset.
- This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor Digital LLC.