| --- |
| license: mit |
| pipeline_tag: depth-estimation |
| tags: |
| - depth-estimation |
| - monocular-depth |
| - zipdepth |
| - edgepad |
| - onnx |
| --- |
| |
| # EdgePad depth models (ZipDepth for mobile GPUs) |
|
|
| This repository mirrors the **PC ONNX** EdgePad depth models published by the |
| **Nightfall team**. The model research, fine-tuning, graph optimization, and |
| exports were created by Nightfall and are credited to its authors and to |
| Fabio Tosi, the author of ZipDepth. |
|
|
| The model-card text below is based closely on the original release notes, |
| with only small changes to describe the files mirrored here. |
|
|
| ## About these models |
|
|
| The monocular depth models Nightfall uses for AI 3D are published for anyone |
| to use: TFLite for phones and headsets (Nightfall runs them on Quest), and |
| ONNX for PCs (Nightfall Meteor runs them there). They are MIT licensed; see |
| [`LICENSE`](LICENSE). |
|
|
| All of them are **EdgePad** builds of |
| [ZipDepth](https://github.com/fabiotosi92/ZipDepth) by Fabio Tosi (MIT): a |
| 6.1M-parameter convolutional network distilled from Depth Anything V2 Large, |
| with no attention or transformer operations, which is what allows it to run |
| entirely on a mobile GPU. There are two generations: |
|
|
| - **Square EdgePad** (shipped in Nightfall v0.7.x): ZipDepth's own Standard |
| checkpoint, **not retrained**, exported through exact graph rewrites so the |
| whole network runs on the GPU. |
| - **Widescreen EdgePad** (Nightfall's next release): a light fine-tune of the |
| same checkpoint for 16:9 video and steadier depth from frame to frame, |
| exported at exact 16:9 sizes. |
|
|
| ## Files in this mirror |
|
|
| | File | Generation | Input | Output | SHA-256 | |
| | --- | --- | --- | --- | --- | |
| | `zipdepth_edgepad_384.onnx` | Square | 384x384 | 384x384 | `78efcf08caded436c626af0b7b5980db47bc07653ec549a59b7e48e73b3e035f` | |
| | `zipdepth_wide_512x288.onnx` | Widescreen | 512x288 | 512x288 | `197d598f50ee3270b12831096dbab260c6bd5979bab13b87f01a28fad0268274` | |
| | `zipdepth_wide_672x384.onnx` | Widescreen | 672x384 | 672x384 | `f812b10b10309c6759266cbe888fa9c0f70668350fdf69e7eda9cc0412ca1a21` | |
|
|
| Sizes are width x height. These are the PC ONNX exports used by Nightfall |
| Meteor. This Hugging Face mirror currently contains the three ONNX files above; |
| the original release also provides TFLite exports for mobile and headset use. |
|
|
| ## Using them |
|
|
| - **Input:** RGB as float32 in 0 to 1 (`pixel / 255`). ImageNet mean and |
| standard deviation are applied inside the graph. The ONNX model takes NCHW |
| `[1, 3, H, W]` named `image`. Resize the frame to the model input size; |
| Nightfall uses an area filter. |
| - **Output:** relative inverse depth: larger is nearer, with no fixed scale or |
| offset. Normalise each frame before use. Nightfall stretches between low and |
| high percentiles and smooths the result over time. The ONNX models output the |
| full-size map, already unpacked and clamped. |
|
|
| ## Square EdgePad: optimisation only |
|
|
| The weights are ZipDepth's Standard checkpoint (`zipdepth_base.pth`), unchanged, |
| including its learned convex reconstruction head, which proved essential for |
| clean full-screen geometry in the headset. Every change in the Square EdgePad |
| export is to the graph and is mathematically exact: |
|
|
| 1. **Materialised attention broadcasts.** ZipDepth's strip, channel, and |
| global-context attention broadcast small tensors across the feature map. |
| These tensors are expanded before the operation for GPU-delegate |
| compatibility. |
| 2. **Small reductions.** Large spatial average pools are split into exact 2x |
| and 3x stages. |
| 3. **Packed reconstruction head.** The head's 5-dimensional softmax and |
| depth-to-space tail becomes four ordinary 9-channel softmax branches, the |
| 36-channel mask convolution becomes four 9-channel convolutions, and the |
| weighted sums become fixed 1x1 convolutions. |
| 4. **EdgePad.** The head's replicate padding is rebuilt from edge slices and |
| concatenation so the whole graph can run without splitting and copying |
| intermediate tensors back to the CPU. |
|
|
| Against the original Standard graph, Nightfall reports correlation |
| `0.9999999999999908` and normalised mean absolute error `3.6e-8`. |
|
|
| ## Widescreen EdgePad: a light fine-tune |
|
|
| One checkpoint (SHA-256 |
| `2dc8fddcef9ef22f45b6a1573b9cf66de0f39b1b5172e4284dd693790bd6309e`) |
| was trained from ZipDepth's Standard checkpoint in two short stages. Only the |
| decoder was trained: 154,005 of 6.1 million parameters. The encoder is |
| ZipDepth's. |
|
|
| - **Stage 1** (5,000 steps): shapes alternated between 384x384, 256x256, |
| 224x224, and 512x288, so one set of weights serves every size. Training used |
| 2,000 frames from 25 videos: gameplay/YouTube, film, live-action/CG, and |
| animation. Labels were the original Square EdgePad-384 output, Video Depth |
| Anything Small for frame-to-frame consistency, and Marigold V2 for spatial |
| detail on selected keyframes. A texture-leakage term at weight 0.5 |
| discourages copying image texture into depth. |
| - **Stage 2** (1,000 steps at learning rate `5e-5`): decoder-only training for |
| the exact 16:9 sizes. |
|
|
| The model has no memory and processes each frame independently; temporal |
| behaviour was learned only through the losses. |
|
|
| Nightfall's held-out video results against the untouched ZipDepth export at |
| the same size: |
|
|
| | Model | Compared with | Flicker on static scenes | Flow-warped error | Frame-to-frame change | |
| | --- | --- | --- | --- | --- | |
| | 512x288 | stock 512x288 | -8.6% | -5.8% | -8.4% | |
| | 320x180 | stock 224x224 | -42.4% | -43.9% | -39.2% | |
|
|
| The trade-off is about 11% higher texture leakage than the stock 224x224 model |
| at 320x180. Nightfall notes that the 320x180 model is also visibly less |
| detailed than the 384x384 model, as its size implies. |
|
|
| Nightfall reports that the TFLite exports match their ONNX sources with |
| correlation `0.99999` and pass LiteRT's GPU compatibility check without |
| warnings. Meteor's ONNX files are the same weights with the 2x2 unpacking and |
| clamp added to the graph and the weights embedded. The 672x384 file is an |
| export of the same checkpoint at a larger size and was not part of the |
| held-out evaluation above. |
|
|
| ## Credits and licences |
|
|
| - **ZipDepth:** Fabio Tosi, MIT. These models are derivatives of its weights; |
| its copyright notice is retained in [`LICENSE`](LICENSE). |
| - **Teacher models** used only to make training labels, not included here: |
| Video Depth Anything Small (Apache-2.0) and Marigold V2 (Apache-2.0; built |
| on Qwen-Image-Edit-2509, Apache-2.0). |
| - **Fine-tuning, graph optimisation, and exports:** Nightfall's model research |
| team. Sincere thanks to **tB0nE and all Nightfall contributors** for making |
| these models and the detailed release notes available. |
|
|
| This Hugging Face repository is a mirror provided for convenient access. All |
| credit for the original models and release documentation belongs to the |
| Nightfall team and Fabio Tosi. |
|
|
