--- license: other license_name: corridor-key-licence license_link: LICENSE library_name: pytorch pipeline_tag: image-segmentation tags: - matting - alpha-matting - green-screen - chroma-key - keying - real-time - obs - corridorkey datasets: - mantelmoritz/CorridorKey-Synthetic-Streamers --- # CorridorKey Real-Time Edge Refiner **A 23k-parameter CNN that brings back native-resolution hair and edge detail when [CorridorKey 2.0](https://huggingface.co/nikopueringer/CorridorKey2.0) runs at reduced resolution for live green-screen keying, e.g. a 4K or 1080p facecam in OBS at 30–60 fps.** ![Input and real-time result on real 4K green-screen footage](images/hero_druid_4k.png) Footage: CorridorKey 1.0 Quality Benchmark (Niko Pueringer / Corridor Digital, Apache-2.0). Real-time pipeline: CorridorKey 2.0 Stage 1 on a 512×512 person crop + this refiner. ## What problem it solves CorridorKey 2.0 produces excellent mattes, but at full 4K it is an offline model. To key live video, the network has to run on a small grid: for example a 512×512 crop around the person, or 512×256 for the whole frame. Its matte then has to be scaled back up to the camera resolution, and that step destroys the edge: - **Bilinear upsampling** gives a blurry, blocky edge with a green or dark halo. - **A colour guided filter** recovers some structure, but it smears hair, leaves fringes and fails where foreground and screen have similar brightness. - **This refiner** looks at the native-resolution pixels and learns how CorridorKey 2.0 would have keyed them at full resolution. It predicts corrections for alpha *and* foreground colour (despill / unmixing) at every native pixel. ![1:1 native-pixel crops on real 4K footage](images/compare_druid_4k.png) Top row: composite over a dark slate. Bottom row: alpha. All crops are 1:1 native pixels, enlarged with nearest neighbour. "CorridorKey 2.0 offline" is the full model at native resolution, the target the refiner was trained on. ## How it works The refiner is a fully convolutional network: 11 input channels, a 3×3 input convolution, four residual dilated 3×3 blocks (dilations 1, 2, 4, 1), GELU, and a 1×1 output convolution. It has 24 channels in v10w and 16 in v9b. Its receptive field is ±9 pixels. Per native pixel it receives: | Channels | Input | |---|---| | 0–2 | `I`: native RGB frame (sRGB) | | 3–5 | `B`: smooth background estimate (sRGB), hole-filled from pixels CorridorKey calls background | | 6–8 | `F`: CorridorKey foreground transferred onto the native frame (`I + upsample(F_low − I_low)`) | | 9 | `a_up`: CorridorKey alpha, bilinear upsampled | | 10 | `a_gf`: alpha of a colour guided filter driven by CorridorKey's low-resolution alpha | It outputs residuals on top of `a_gf` (alpha) and `F` (foreground). The result is used only inside an uncertainty band around CorridorKey's edges. Outside the band the output is the colour-guided upsampling of CorridorKey's matte, which in solid foreground and clean background is CorridorKey's own 0 or 1. The band is weighted by nearby foreground, which keeps the refiner out of dark or noisy screen areas. In the real-time engine the refiner only runs on 32×32 tiles that intersect the band, with a fixed tile budget (most uncertain tiles first). In fp16 that costs about 5–6 ms per frame with 256 tiles and 10–11 ms with 512 tiles on an RTX 5070 Ti. `corridorkey_refiner.py` in this repository is a standalone full-frame reference implementation of the same maths. On a test frame it matches the tiled engine to a mean absolute difference of 0.0001 in alpha. ## Files | File | Description | |---|---| | `corridorkey_edge_refiner_v10w.safetensors` | **recommended**, 24 channels, 23,332 parameters | | `corridorkey_edge_refiner_v9b.safetensors` | 16 channels, 10,948 parameters, slightly cheaper, slightly worse | | `corridorkey_refiner.py` | model definition, loader, input construction and full-frame `refine()` | | `LICENSE` | Corridor Key Licence, Version 2.0 | ## Usage You need CorridorKey 2.0 (code and weights from https://huggingface.co/nikopueringer/CorridorKey2.0, not included here) to produce the low-resolution matte. Then: ```python import torch from corridorkey_refiner import load_refiner, refine net = load_refiner("corridorkey_edge_refiner_v10w.safetensors") # fp16, CUDA # rgb: (1, 3, H, W) native frame, sRGB in [0, 1] # rgb_low: (1, 3, h, w) the (crop of the) frame as CorridorKey saw it # alpha_low: (1, 1, h', w') CorridorKey alpha at its output resolution (e.g. Stage-1 aux head, 256x256) # fg_low: (1, 3, h', w') CorridorKey straight foreground, sRGB alpha, fg = refine(net, rgb, rgb_low, alpha_low, fg_low) # native resolution ``` If you key a crop, pass the native-resolution crop as `rgb` and paste the result back. Apply despill and alpha contrast after the refiner, as you would for CorridorKey's own output. ## Results Edge error = mean absolute alpha error inside the edge band (×1000), measured against CorridorKey 2.0 offline at native resolution; flicker = temporal error of the frame-to-frame alpha change in the edge band (×1000). Lower is better. 32 consecutive frames per segment. None of these clips were used for training. - **Real stock footage:** 6 Pexels / Mixkit green-screen clips (4K, 4K DCI, 1080p, 720p), curly / wavy / long hair, headphones, headset, glasses. - **Real 4K, hair** and **Real 4K, transparency:** the two clips of the CorridorKey 1.0 Quality Benchmark (2 segments each). The druid has curly hair, a beard and fur; the clown has tulle and water glasses. - **Generated streamers:** the 10 test clips of series 3 of the dataset. They are deliberately hard (props in front of the person, partially covered screens, people leaving the frame). **Person crop, 512×512** (CorridorKey 2.0 Stage 1 on a tracked square around the person): | Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame | |---|---|---|---|---|---|---| | Bilinear | 121.3 | 91.2 | 80.1 | 144.7 | 75.3 | — | | Colour guided filter | 51.2 | 45.3 | 64.1 | 116.8 | 33.1 | 14.4 | | Edge Refiner v9b | 32.5 | 32.7 | **61.9** | 104.6 | 22.0 | 24.4 | | **Edge Refiner v10w** | **30.9** | **30.6** | 62.3 | **101.2** | **20.5** | 25.8 | **Full frame, 512×256** (cheaper, for 60 fps): | Upsampling | Real stock | Real 4K, hair | Real 4K, transparency | Generated streamers | Flicker (real stock) | GPU ms / frame | |---|---|---|---|---|---|---| | Bilinear | 174.7 | 117.8 | 106.7 | 190.1 | 91.5 | — | | Colour guided filter | 61.3 | 53.4 | 76.4 | 144.0 | 34.9 | 11.8 | | Edge Refiner v9b | 41.7 | 46.3 | 73.6 | 135.9 | 26.4 | 16.5 | | **Edge Refiner v10w** | **39.4** | **45.2** | **73.5** | **133.4** | **25.0** | 17.6 | GPU time is end-to-end per frame, CorridorKey 2.0 Stage 1 included, on an RTX 5070 Ti. It is the median over the 6 real stock clips, measured without video I/O. The refiner itself costs about 5–6 ms with a budget of 256 tiles and 10–11 ms with 512 tiles. The tile count, not the frame resolution, dominates the cost. - Compared with the guided filter, the refiner (v10w) reduces the edge error by 36–40 % on the real stock clips and by 15–32 % on the 4K druid clip. Flicker drops by 28–38 %. - On hair (druid) the gain is largest. On semi-transparent material (clown) it is small: there the result is limited by what the low-resolution network sees. ![Generated streamer clip from the test split, 1:1 crops](images/compare_generated_streamer.png) ## Training - **Target:** CorridorKey 2.0 offline at native resolution (alpha and foreground). The refiner is a student of CorridorKey: it cannot be sharper than its teacher, and it inherits the teacher's mistakes. - **Inputs during training:** exactly what the real-time engine produces at run time. Training patches were cut by running the real-time pipeline (512×512 person crop and 512×256 full frame, Stage 1 only) over every training clip and computing the 11 input channels, mostly from the edge band. - **Data:** - 32 segments of real stock green-screen footage (Pexels, Mixkit) - the same frames re-composited from the teacher's foreground and alpha over procedural green/teal screens, also passed through a "bad camera" model (sensor noise, Bayer demosaicing, 4:2:0 chroma, JPEG, exposure errors) - 196 clips of the [CorridorKey Synthetic Streamers](https://huggingface.co/datasets/mantelmoritz/CorridorKey-Synthetic-Streamers) dataset, without real-room clips and without clips where the teacher keys green clothing away Real footage is sampled 3× more often than its share. 110,000 patches, 24,000 steps, loss on alpha, alpha gradients, composite and foreground. - **Evaluation data was never used for training.** ## Limitations ![Limitation: free-flying strands far from the head](images/compare_facecam_1080.png) - **The teacher is the ceiling.** Where CorridorKey 2.0 is soft at native resolution, the refiner is soft too. - **It can only refine what the low-resolution network found.** Single strands far outside CorridorKey's low-resolution matte are outside the refiner band and are lost, as in the image above. Semi-transparent objects (glass, tulle, water) depend almost entirely on the low-resolution network. - Trained for **green (and teal / blue-green) screens** with people in front, mostly facecam and upper-body framing. It is not a general background-removal model. - The numbers above compare against CorridorKey 2.0, not against hand-made ground truth. ## License **Corridor Key Licence, Version 2.0** (CC BY-NC-SA 4.0 with Additional Terms by Corridor Digital LLC). The full text is in [`LICENSE`](LICENSE). This refiner was trained on the outputs of CorridorKey 2.0 and only works together with it. It is therefore released as a CorridorKey derivative under the same licence and keeps the CorridorKey name (Additional Terms §4, §5). In short: - **Free of charge.** You may not sell it, repackage it as a standalone or competing product, or offer it as a paid API or inference service. - **Share-alike:** derivatives must be released under the same licence, free of charge, and keep the "CorridorKey" name. - Images and video you produce with it are yours. - Commercial software integrations require a separate agreement with Corridor Digital (contact@corridordigital.com). Organisations above US $20M revenue need CorridorKey plugin seats (https://corridorkey.com). ### Attribution and modifications - **CorridorKey 2.0** © Corridor Digital LLC. The model was designed and trained by Zachary Harris in collaboration with Niko Pueringer and Corridor Digital. https://huggingface.co/nikopueringer/CorridorKey2.0 · https://corridorkey.com - CorridorKey 2.0 builds on frozen foundation models under their own licences (MoGe-2 by Microsoft, RVM by Google DeepMind, optional C-RADIO by NVIDIA). None of them, and no CorridorKey weights, are contained in this repository. - **Changes relative to CorridorKey 2.0:** this repository adds a new, separately trained edge-refinement network and its input pipeline. The CorridorKey 2.0 weights and code are not modified or redistributed. - Example footage: CorridorKey 1.0 Quality Benchmark (https://huggingface.co/datasets/nikopueringer/CorridorKey1.0_QualityBenchmark, Apache-2.0), Pexels (Pexels licence), and the CorridorKey Synthetic Streamers dataset. - This is an independent community project. It is not affiliated with, sponsored by or endorsed by Corridor Digital LLC.