Add EdgePad ONNX models and original model card
Browse files- LICENSE +23 -0
- README.md +140 -3
- zipdepth_edgepad_384.onnx +3 -0
- zipdepth_wide_512x288.onnx +3 -0
- zipdepth_wide_672x384.onnx +3 -0
LICENSE
CHANGED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Fabio Tosi (ZipDepth, the weights these models derive from)
|
| 4 |
+
Copyright (c) 2026 The Nightfall authors (the fine-tuned weights and the
|
| 5 |
+
exported graphs)
|
| 6 |
+
|
| 7 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 8 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 9 |
+
in the Software without restriction, including without limitation the rights
|
| 10 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 11 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 12 |
+
furnished to do so, subject to the following conditions:
|
| 13 |
+
|
| 14 |
+
The above copyright notice and this permission notice shall be included in all
|
| 15 |
+
copies or substantial portions of the Software.
|
| 16 |
+
|
| 17 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 18 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 19 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 20 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 21 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 22 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 23 |
+
SOFTWARE.
|
README.md
CHANGED
|
@@ -1,5 +1,142 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
| 3 |
-
|
| 4 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
license: mit
|
| 3 |
+
pipeline_tag: depth-estimation
|
| 4 |
+
tags:
|
| 5 |
+
- depth-estimation
|
| 6 |
+
- monocular-depth
|
| 7 |
+
- zipdepth
|
| 8 |
+
- edgepad
|
| 9 |
+
- onnx
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
# EdgePad depth models (ZipDepth for mobile GPUs)
|
| 13 |
+
|
| 14 |
+
This repository mirrors the **PC ONNX** EdgePad depth models published by the
|
| 15 |
+
**Nightfall team**. The model research, fine-tuning, graph optimization, and
|
| 16 |
+
exports were created by Nightfall and are credited to its authors and to
|
| 17 |
+
Fabio Tosi, the author of ZipDepth.
|
| 18 |
+
|
| 19 |
+
The model-card text below is based closely on the original release notes,
|
| 20 |
+
with only small changes to describe the files mirrored here.
|
| 21 |
+
|
| 22 |
+
## About these models
|
| 23 |
+
|
| 24 |
+
The monocular depth models Nightfall uses for AI 3D are published for anyone
|
| 25 |
+
to use: TFLite for phones and headsets (Nightfall runs them on Quest), and
|
| 26 |
+
ONNX for PCs (Nightfall Meteor runs them there). They are MIT licensed; see
|
| 27 |
+
[`LICENSE`](LICENSE).
|
| 28 |
+
|
| 29 |
+
All of them are **EdgePad** builds of
|
| 30 |
+
[ZipDepth](https://github.com/fabiotosi92/ZipDepth) by Fabio Tosi (MIT): a
|
| 31 |
+
6.1M-parameter convolutional network distilled from Depth Anything V2 Large,
|
| 32 |
+
with no attention or transformer operations, which is what allows it to run
|
| 33 |
+
entirely on a mobile GPU. There are two generations:
|
| 34 |
+
|
| 35 |
+
- **Square EdgePad** (shipped in Nightfall v0.7.x): ZipDepth's own Standard
|
| 36 |
+
checkpoint, **not retrained**, exported through exact graph rewrites so the
|
| 37 |
+
whole network runs on the GPU.
|
| 38 |
+
- **Widescreen EdgePad** (Nightfall's next release): a light fine-tune of the
|
| 39 |
+
same checkpoint for 16:9 video and steadier depth from frame to frame,
|
| 40 |
+
exported at exact 16:9 sizes.
|
| 41 |
+
|
| 42 |
+
## Files in this mirror
|
| 43 |
+
|
| 44 |
+
| File | Generation | Input | Output | SHA-256 |
|
| 45 |
+
| --- | --- | --- | --- | --- |
|
| 46 |
+
| `zipdepth_edgepad_384.onnx` | Square | 384x384 | 384x384 | `78efcf08caded436c626af0b7b5980db47bc07653ec549a59b7e48e73b3e035f` |
|
| 47 |
+
| `zipdepth_wide_512x288.onnx` | Widescreen | 512x288 | 512x288 | `197d598f50ee3270b12831096dbab260c6bd5979bab13b87f01a28fad0268274` |
|
| 48 |
+
| `zipdepth_wide_672x384.onnx` | Widescreen | 672x384 | 672x384 | `f812b10b10309c6759266cbe888fa9c0f70668350fdf69e7eda9cc0412ca1a21` |
|
| 49 |
+
|
| 50 |
+
Sizes are width x height. These are the PC ONNX exports used by Nightfall
|
| 51 |
+
Meteor. This Hugging Face mirror currently contains the three ONNX files above;
|
| 52 |
+
the original release also provides TFLite exports for mobile and headset use.
|
| 53 |
+
|
| 54 |
+
## Using them
|
| 55 |
+
|
| 56 |
+
- **Input:** RGB as float32 in 0 to 1 (`pixel / 255`). ImageNet mean and
|
| 57 |
+
standard deviation are applied inside the graph. The ONNX model takes NCHW
|
| 58 |
+
`[1, 3, H, W]` named `image`. Resize the frame to the model input size;
|
| 59 |
+
Nightfall uses an area filter.
|
| 60 |
+
- **Output:** relative inverse depth: larger is nearer, with no fixed scale or
|
| 61 |
+
offset. Normalise each frame before use. Nightfall stretches between low and
|
| 62 |
+
high percentiles and smooths the result over time. The ONNX models output the
|
| 63 |
+
full-size map, already unpacked and clamped.
|
| 64 |
+
|
| 65 |
+
## Square EdgePad: optimisation only
|
| 66 |
+
|
| 67 |
+
The weights are ZipDepth's Standard checkpoint (`zipdepth_base.pth`), unchanged,
|
| 68 |
+
including its learned convex reconstruction head, which proved essential for
|
| 69 |
+
clean full-screen geometry in the headset. Every change in the Square EdgePad
|
| 70 |
+
export is to the graph and is mathematically exact:
|
| 71 |
+
|
| 72 |
+
1. **Materialised attention broadcasts.** ZipDepth's strip, channel, and
|
| 73 |
+
global-context attention broadcast small tensors across the feature map.
|
| 74 |
+
These tensors are expanded before the operation for GPU-delegate
|
| 75 |
+
compatibility.
|
| 76 |
+
2. **Small reductions.** Large spatial average pools are split into exact 2x
|
| 77 |
+
and 3x stages.
|
| 78 |
+
3. **Packed reconstruction head.** The head's 5-dimensional softmax and
|
| 79 |
+
depth-to-space tail becomes four ordinary 9-channel softmax branches, the
|
| 80 |
+
36-channel mask convolution becomes four 9-channel convolutions, and the
|
| 81 |
+
weighted sums become fixed 1x1 convolutions.
|
| 82 |
+
4. **EdgePad.** The head's replicate padding is rebuilt from edge slices and
|
| 83 |
+
concatenation so the whole graph can run without splitting and copying
|
| 84 |
+
intermediate tensors back to the CPU.
|
| 85 |
+
|
| 86 |
+
Against the original Standard graph, Nightfall reports correlation
|
| 87 |
+
`0.9999999999999908` and normalised mean absolute error `3.6e-8`.
|
| 88 |
+
|
| 89 |
+
## Widescreen EdgePad: a light fine-tune
|
| 90 |
+
|
| 91 |
+
One checkpoint (SHA-256
|
| 92 |
+
`2dc8fddcef9ef22f45b6a1573b9cf66de0f39b1b5172e4284dd693790bd6309e`)
|
| 93 |
+
was trained from ZipDepth's Standard checkpoint in two short stages. Only the
|
| 94 |
+
decoder was trained: 154,005 of 6.1 million parameters. The encoder is
|
| 95 |
+
ZipDepth's.
|
| 96 |
+
|
| 97 |
+
- **Stage 1** (5,000 steps): shapes alternated between 384x384, 256x256,
|
| 98 |
+
224x224, and 512x288, so one set of weights serves every size. Training used
|
| 99 |
+
2,000 frames from 25 videos: gameplay/YouTube, film, live-action/CG, and
|
| 100 |
+
animation. Labels were the original Square EdgePad-384 output, Video Depth
|
| 101 |
+
Anything Small for frame-to-frame consistency, and Marigold V2 for spatial
|
| 102 |
+
detail on selected keyframes. A texture-leakage term at weight 0.5
|
| 103 |
+
discourages copying image texture into depth.
|
| 104 |
+
- **Stage 2** (1,000 steps at learning rate `5e-5`): decoder-only training for
|
| 105 |
+
the exact 16:9 sizes.
|
| 106 |
+
|
| 107 |
+
The model has no memory and processes each frame independently; temporal
|
| 108 |
+
behaviour was learned only through the losses.
|
| 109 |
+
|
| 110 |
+
Nightfall's held-out video results against the untouched ZipDepth export at
|
| 111 |
+
the same size:
|
| 112 |
+
|
| 113 |
+
| Model | Compared with | Flicker on static scenes | Flow-warped error | Frame-to-frame change |
|
| 114 |
+
| --- | --- | --- | --- | --- |
|
| 115 |
+
| 512x288 | stock 512x288 | -8.6% | -5.8% | -8.4% |
|
| 116 |
+
| 320x180 | stock 224x224 | -42.4% | -43.9% | -39.2% |
|
| 117 |
+
|
| 118 |
+
The trade-off is about 11% higher texture leakage than the stock 224x224 model
|
| 119 |
+
at 320x180. Nightfall notes that the 320x180 model is also visibly less
|
| 120 |
+
detailed than the 384x384 model, as its size implies.
|
| 121 |
+
|
| 122 |
+
Nightfall reports that the TFLite exports match their ONNX sources with
|
| 123 |
+
correlation `0.99999` and pass LiteRT's GPU compatibility check without
|
| 124 |
+
warnings. Meteor's ONNX files are the same weights with the 2x2 unpacking and
|
| 125 |
+
clamp added to the graph and the weights embedded. The 672x384 file is an
|
| 126 |
+
export of the same checkpoint at a larger size and was not part of the
|
| 127 |
+
held-out evaluation above.
|
| 128 |
+
|
| 129 |
+
## Credits and licences
|
| 130 |
+
|
| 131 |
+
- **ZipDepth:** Fabio Tosi, MIT. These models are derivatives of its weights;
|
| 132 |
+
its copyright notice is retained in [`LICENSE`](LICENSE).
|
| 133 |
+
- **Teacher models** used only to make training labels, not included here:
|
| 134 |
+
Video Depth Anything Small (Apache-2.0) and Marigold V2 (Apache-2.0; built
|
| 135 |
+
on Qwen-Image-Edit-2509, Apache-2.0).
|
| 136 |
+
- **Fine-tuning, graph optimisation, and exports:** Nightfall's model research
|
| 137 |
+
team. Sincere thanks to **tB0nE and all Nightfall contributors** for making
|
| 138 |
+
these models and the detailed release notes available.
|
| 139 |
+
|
| 140 |
+
This Hugging Face repository is a mirror provided for convenient access. All
|
| 141 |
+
credit for the original models and release documentation belongs to the
|
| 142 |
+
Nightfall team and Fabio Tosi.
|
zipdepth_edgepad_384.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:78efcf08caded436c626af0b7b5980db47bc07653ec549a59b7e48e73b3e035f
|
| 3 |
+
size 24608261
|
zipdepth_wide_512x288.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:197d598f50ee3270b12831096dbab260c6bd5979bab13b87f01a28fad0268274
|
| 3 |
+
size 24625797
|
zipdepth_wide_672x384.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f812b10b10309c6759266cbe888fa9c0f70668350fdf69e7eda9cc0412ca1a21
|
| 3 |
+
size 24624796
|