ShadeNet-6 L

Single-image inverse rendering — one RGB photo in, albedo + relative depth + surface normal out.

ShadeNet-6 L outputs on held-out images

A feed-forward network on a frozen DINOv2 ViT-B/14 backbone (86.6M parameters) plus a 11.8M-parameter trainable head: a projector, a convolutional detail encoder that sees the photo at full resolution (DINOv2's maps are smooth, it reads 14-pixel patches), a wide decoder, a small dense patch network that gives a coarse first guess, an input patch dictionary and a tiny refiner. One forward pass, no iteration. On the 23-image held-out set its single pass has a 26% lower mean error than ShadeNet-5's single pass and is better on every one of the 23 images (see Results for the caveats).

Hero: the first six validation images, in dataset order, not picked, single pass plus the optional separate refiner. Each panel: input | albedo / depth | normal.

TL;DR

Task Single-image inverse rendering (intrinsic decomposition)
Output 7ch — albedo (3) + relative depth (1) + normal (3)
Params 98.35M total (11.77M trainable; 86.58M is the frozen DINOv2 ViT-B/14)
Backbone DINOv2 ViT-B/14, frozen (Meta AI)
Inference one forward pass (about 39 ms per 448x448 image on an RTX 5060 Ti, bf16); an optional separate refiner (0.9M, one cheap extra pass) and a mirror average (2 passes) each help a little
Input size 448x448 (multiples of 14 and 32: 448, 896); inference/run.py keeps the whole photo
Framework PyTorch (+ transformers); standalone inference code in inference/, no training code needed; int8 (103 MB) and int4 (55 MB) ONNX exports for ONNX Runtime, each also with the refiner built in (*_refined.onnx)
License Apache-2.0

What it does

Given a single RGB image it predicts three intrinsic maps: albedo (base colour), relative depth (grayscale, per-image) and a surface normal map. Output channels are [0:3]=albedo, [3:4]=depth, [4:7]=normal, all in [-1, 1]; depth is relative (larger = farther), normals are x right, y up, z toward the viewer.

Same output convention as ShadeNet-5: 7 channels, no shading channel. It is a new architecture trained from scratch, not a continuation of ShadeNet-5's weights.

Results

Mean per-map L1 against the labels on the same 23 held-out images, run through both models in one script (lower is better; albedo and normal are scored on labelled pixels only, depth on all pixels; unit-length normals in the "+unit" rows). ShadeNet-5 numbers are measured here on the same images, not copied from its card (the ShadeNet-5 card's 0.1757 is on a different set of 23 images).

Model and recipe Parameters Albedo Depth Normal Mean
ShadeNet-5, single pass + unit normals 34.65M 0.166 0.169 0.163 0.1659
ShadeNet-5 + its refiner 34.65M + 0.91M 0.160 0.168 0.151 0.1597
ShadeNet-5, six passes (mirror + rotations ±4°) 34.65M 0.158 0.162 0.150 0.1562
ShadeNet-6 L, single pass + unit normals 98.35M 0.123 0.124 0.122 0.1232
ShadeNet-6 L, single pass (raw normals) 98.35M 0.123 0.124 0.126 0.1244
ShadeNet-6 L, mirror average 98.35M 0.122 0.122 0.119 0.1213
ShadeNet-6 L, six passes (mirror + rotations ±4°) 98.35M 0.120 0.129 0.118 0.1223
ShadeNet-6 L + separate refiner, single pass 98.35M + 0.91M 0.121 0.124 0.121 0.1223
ShadeNet-6 L, mirror average, then the refiner 98.35M + 0.91M 0.120 0.122 0.119 0.1207
ShadeNet-6 XS, single pass + unit normals 1.99M 0.176 0.186 0.166 0.1759

Paired per-image difference in mean L1 (ShadeNet-6 L minus ShadeNet-5; negative = ShadeNet-6 L is better):

Comparison Difference ShadeNet-6 L better on
single pass vs single pass -0.0427 ± 0.0055 23 of 23 images
single pass vs ShadeNet-5 + refiner -0.0364 ± 0.0055 23 of 23
mirror average vs ShadeNet-5 + refiner -0.0384 ± 0.0056 23 of 23
six passes vs ShadeNet-5 six passes -0.0340 ± 0.0053 22 of 23
  • The gain is on all three maps: albedo 0.166 to 0.123, depth 0.169 to 0.124, normals 0.163 to 0.122.
  • Test-time averaging barely helps this model: a mirror average gains 0.002 and six passes make depth worse (0.124 to 0.129), where ShadeNet-5 gains 0.010 from its six passes. Use a single pass or the mirror average.
  • The model's own tiny refiner (0.51M) adds 0.0001 to 0.0017 on top of the main network's maps.
  • A separate refiner (below) adds a further 0.0010 (single pass, better on 18 of 23 images) or 0.0026 together with the mirror average (better on 22 of 23, 0.1207): small, but it stacks with the mirror average.
  • Speed: one pass takes about 39 ms per image on an RTX 5060 Ti (bf16). ShadeNet-5's six-pass recipe took about 283 ms per image in the same run.

Single pass of ShadeNet-5 and ShadeNet-6 L on the same validation photos

Six of the 23 validation photos, chosen by the size of the difference, so the figure is not cherry-picked: the two where ShadeNet-6 L gains most, two typical ones, and the two where it gains least. Columns: photo, then albedo, depth and normal as label | ShadeNet-5 | ShadeNet-6 L (the column headers say "ShadeNet-6"). Both models: one pass, unit-length normals. The number on each photo is its mean L1 for each model.

Read these numbers with care. The validation set is 23 images, every label is a Marigold-V2 pseudo-label (not a measurement), and the checkpoint was chosen on these same 23 images, which makes the score optimistic by roughly 0.001 to 0.003. The gap to ShadeNet-5 (0.04) is more than ten times that, but it is one run on a small set. See Limitations.

Gallery

Held-out images (none was used for training), single forward pass plus the separate refiner, unit-length normals. These are the first eight of the 23 validation images in dataset order, not selected for how they look. Each panel: input | albedo / depth | normal.

ski jump in a snow park

ski jump

man on a bench behind a bicycle

bench and bicycle

fairground wheel with spectators

fairground wheel

climber leaping from a boulder

boulder

child in an orange tube slide

tube slide

child hanging upside down from bars

monkey bars

boy swinging on a pole

pole swing

portable toilets and city towers

towers

Separate refiner (optional)

refiner/refiner.pt is a 0.91M-parameter map refiner (the same small 4-level UNet as ShadeNet-5's, width 32) that runs after the model: it takes the photo and the model's 7 clamped maps and returns corrected maps. It is optional; the model's own 0.51M refiner is already inside the checkpoint, and this one sits on top of it.

How it was trained (no labels). For 8,016 photos from the Open Images V7 validation split (only photos listed as CC BY 2.0; attribution for every photo in refiner/attribution.csv; the photos are not redistributed), the frozen model made its single pass and a better teacher map: albedo and normals from the mean of six passes (mirror x rotations 0, ±4°) and a pass at 896 px (photo upsampled 2x, predicted, area-downsampled), depth from the mirror average only, because rotations make this model's depth worse. The refiner learns to turn the single pass into the teacher in one step (L1, random 320 px crops, mirror with normal x negated, 6,000 steps, batch 16, about 35 minutes). On 100 held-out Open Images photos its distance to the teacher fell from 0.0338 to 0.0267 (normals 0.0491 to 0.0316, albedo 0.0313 to 0.0272, depth unchanged).

What it buys, against the true labels (the same 23 images; mean L1, lower is better):

Recipe Albedo Depth Normal Mean Paired change vs single pass
single pass + unit normals 0.1234 0.1242 0.1221 0.1232 n/a
+ refiner 0.1214 0.1242 0.1212 0.1223 -0.0010 ± 0.0002 (18 of 23 images)
refiner applied twice 0.1210 0.1242 0.1220 0.1224 -0.0008
mirror average 0.1221 0.1224 0.1194 0.1213 -0.0019 ± 0.0003 (21 of 23)
mirror average, then refiner 0.1204 0.1224 0.1192 0.1207 -0.0026 ± 0.0004 (22 of 23)
the teacher (7 passes, what it imitates) 0.1184 0.1224 0.1174 0.1194 -0.0038 ± 0.0005
  • The gain is small. The refiner recovers about a quarter of the teacher's gain (ShadeNet-5's refiner recovered about two thirds of its six-pass gain), because this model's single pass is already close to its own ensemble. It improves albedo and, less, normals; it does not change depth.
  • Mirror average alone beats the refiner alone (0.1213 against 0.1223) for a second forward pass of the big model; the refiner costs far less. The two together are the best recipe measured (0.1207).
  • A refiner trained on the true labels instead (the main model frozen, on the Flickr8k training photos) did not improve in its first two epochs (0.1238, 0.1241 against 0.1239) and was stopped. It is not a full test: the main model has seen those photos, so its outputs on them are cleaner than on new photos and leave the refiner little to correct.

Single pass, refined, and the change against the true label, for six validation images

Six validation images (the first six, not picked). For each map: single pass | refined | gain against the true label (per-pixel |single - label| minus |refined - label|; green = the refiner is closer, red = further, grey = unchanged; full colour at ±0.05). Depth is grey everywhere: the refiner does not change it.

Use it with python inference/run.py photos/ --checkpoint checkpoints/shadenet6-l.ckpt --refine (add --mirror for the best recipe); the ONNX files named *_refined.onnx have it built in, and onnx/shadenet6_l_refiner.onnx (3.7 MB) is the refiner alone, to chain after the plain ONNX files.

Quantised ONNX (int8)

The weights are 394 MB in fp32. onnx/shadenet6_l_int8.onnx (103 MB) is the same model with int8 weights and gives the same score: mean L1 0.1233 against 0.1234 for the fp32 model, on the same 23 images. onnx/shadenet6_l_int4.onnx (55 MB) has int4 weights: the smallest option, but it loses accuracy (mean L1 0.1311, +0.0077; albedo suffers most). Both are for storage and download size only: neither is faster (see the timing note below). Each also exists with the separate refiner built in: onnx/shadenet6_l_int8_refined.onnx (107 MB, mean L1 0.1223) and onnx/shadenet6_l_int4_refined.onnx (59 MB, mean L1 0.1299, against 0.1311 without the refiner); same input and output, 1.4 s and 2.1 s per image on the CPU (8 threads).

Method: weight-only, symmetric, block-wise along the input axis (blocks of up to 128), the scale of each block searched to minimise the squared error, fp16 scales, a DequantizeLinear node (opset 21) in front of every large weight; the other large tensors are stored in fp16. The graph is fixed at 448x448: input input_rgb [1,3,448,448] in [-1, 1], output maps [1,7,448,448], clamped, with unit-length normals inside.

Which layers can be squeezed. First a PyTorch study (fake quantisation: quantise and dequantise in place, one layer group at a time, everything else fp32, mean L1 on the 23 images; change against fp32, lower is better):

group (weights) 8 bit 6 bit 4 bit 3 bit
ViT attention q/k/v (21.2M) -0.0001 0.0000 +0.0012 +0.0144
ViT attention output (7.1M) -0.0001 -0.0003 +0.0007 +0.0030
ViT MLP fc1 (28.3M) +0.0001 +0.0004 +0.0008 +0.0128
ViT MLP fc2 (28.3M) +0.0001 +0.0003 +0.0023 +0.0135
backbone projector 0.0000 0.0000 -0.0002 +0.0005
pre-refiner 0.0000 +0.0001 +0.0007 +0.0023
detail encoder + stems 0.0000 -0.0001 +0.0007 +0.0022
context block 0.0000 0.0000 0.0000 +0.0001
decoder 1x1 convolutions 0.0000 +0.0001 +0.0013 +0.0032
refiner 0.0000 0.0000 0.0000 0.0000

Every group at one width: 8 bit -0.0001, 6 bit +0.0004, 4 bit +0.0076, 3 bit +0.1162 (the model falls apart). The ViT's linear layers are the fragile part at 3 bits; the projector, the context block and the refiner are almost free even at 3 bits, but together they are only 7.5M of the 98M weights, so they barely change the file size. ONNX can only store int4 and int8, so 6 bit is not exportable.

Then the real exports, run with ONNX Runtime on the CPU (the simulation was optimistic for the mix):

export size albedo depth normal mean L1 mean abs. difference to fp32 time per image
fp32 393.8 MB 0.1239 0.1241 0.1220 0.1234 0 0.80 s
int8 (shipped) 103.2 MB 0.1236 0.1243 0.1219 0.1233 0.0023 1.12 s
int4 attention + fc1, int8 elsewhere 74.4 MB 0.1289 0.1236 0.1235 0.1253 0.0286 1.28 s
int4 everywhere 55.4 MB 0.1404 0.1252 0.1276 0.1311 0.0468 1.33 s
  • int8 changes nothing measurable: the output differs from fp32 by 0.002 on average. Use this one unless size matters more than accuracy.
  • int4 hurts albedo first (0.1239 to 0.1289 for the mix, 0.1404 for all-int4). The mix looks nearly free on mean L1 (+0.0019) but moves the output by about 0.03 per pixel on average, so it is not shipped. int4 everywhere is shipped as onnx/shadenet6_l_int4.onnx (+0.0077 mean L1, albedo 0.1404 against 0.1239): a measurable loss, use it only when 55 MB matters.
  • Speed: the quantised files are 40% to 65% slower than fp32 on the CPU, because the weights are expanded back to full precision as the graph runs. Time is the median of three runs of one 448x448 image with 8 threads on a shared cloud CPU slice, so treat it as indicative. If you want speed, use the PyTorch checkpoint on a GPU (about 39 ms per image).
  • onnx/shadenet6_l_refiner.onnx (3.7 MB) is the refiner by itself: inputs photo [1,3,448,448] in [-1, 1] and maps [1,7,448,448] (a model's maps), output refined [1,7,448,448], clamped, normals unit length. Chained after the plain files it gives the same scores as the merged ones (plain int8 + refiner 0.1223, plain int4 + refiner 0.1299), so it is the way to add the refiner to a plain file you already have:
    import numpy as np, onnxruntime as ort
    main = ort.InferenceSession("onnx/shadenet6_l_int8.onnx")
    refiner = ort.InferenceSession("onnx/shadenet6_l_refiner.onnx")
    maps = main.run(None, {"input_rgb": photo})[0]
    refined = refiner.run(None, {"photo": photo, "maps": maps})[0]
    
  • The *_refined.onnx files are built from a separate export that leaves the normals as they come out of the network (the refiner was trained on those), chain the refiner after it, then clamp and renormalise the normals; the int8 one matches the PyTorch result with the refiner exactly (0.1223).
  • Tested with ONNX Runtime 1.32 (opset 21 DequantizeLinear needs a recent version). The inference/run.py script picks up the file with --checkpoint onnx/shadenet6_l_int8.onnx.

Architecture

Three networks in a row; a stop-gradient sits between each pair, so every part is trained on its own loss.

# Part Parameters What it does
1 Pre-refiner 362,624 A dense (fully connected) network applied to every 16x16 patch of the photo with shared weights: 768 numbers in, 1,792 out (a 16x16x7 patch of maps). Gives a coarse first guess.
2 Main model 97,476,825 A frozen DINOv2 ViT-B/14 (86,580,480; facebook/dinov2-base; the outputs of four blocks, 3/6/9/12, are tapped and concatenated) with a 1x1 projector per pyramid level (2,753,408, trainable); a convolutional detail encoder at full resolution (strides 2 to 16, widths 48/96/192/320); a depthwise-separable decoder (widths 1024/512/320/160/96, two blocks per level) with a global-context block; and one soft patch dictionary (33,138 parameters) at its input. Decoder, encoder and dictionary together: about 8.14M. Takes the photo plus the pre-refiner's maps and outputs those maps plus a learned residual.
3 Refiner 511,903 A small 4-level convolutional UNet on the main model's clamped maps. The final output.

There is no post-refiner in this model. A small PatchGAN discriminator (73,697 parameters, GroupNorm) is used in training only and is not in the released checkpoint.

The ViT never resizes its input, so height and width must be multiples of 14 and 32: 448 is what it was trained at (a 32x32 token grid). DINOv2's patch tokens are smooth at 14 pixels; the detail encoder is what supplies the high-resolution structure, as ShadeNet-5's UNet encoder did.

Training

  • Data: Flickr8k photos with Marigold-V2 pseudo-labels (albedo, normals, depth), 8,013 training and 23 validation scenes after a quality gate on the normal labels (41 scenes dropped). 448 px, with zoom, small rotation, warp and mirror augmentation (normals are reoriented with the geometry), plus photometric jitter on the photo only.
  • Loss: per-map task losses on each part's own output — albedo: masked L1 + multiscale gradient matching; depth: scale-and-shift-invariant L1 + gradient matching + a raw-L1 anchor; normal: mean angle + a unit-length penalty — plus a depth-orientation term: the scale-and-shift losses cannot tell a depth map from its inverse, and from scratch the model settled into inverted depth until relu(-scale) was added. In phase 2 a PatchGAN term (weight 0.02, discriminator lr 2e-5, ramped in over 1,000 steps) judges the main model's albedo and normal maps; depth stays on L1.
  • Optimiser: AdamW with warm-up and cosine decay, per-part gradient clipping, bf16, batch 8. About 10 minutes per epoch on one RTX 5060 Ti.
  • Phase 1 (L1 only): peak learning rate 3e-4 on a 40-epoch cosine. Mean L1 on the training-time validation: epoch 0 0.1647, epoch 3 0.1428, best epoch 6 0.1364. Stopped by hand after epoch 7.
  • Phase 2 (+ PatchGAN): restarted from the phase-1 best with a fresh optimiser, peak learning rate 1e-4 on a 25-epoch cosine. Epoch 0 0.1314, best epoch 9 0.1243, epochs 9 to 14 between 0.1243 and 0.1275 (a plateau at about 0.125). Stopped by hand after epoch 14. The released checkpoint is epoch 9.
  • The frozen ViT is held in eval mode inside the forward pass (Lightning restores each module's saved mode after its sanity check, which would otherwise switch it back). The pre-refiner, main model and refiner share one optimiser but no loss.

What we learned

  • The larger frozen backbone, the detail encoder and the wider decoder together were the big step (0.176 for the 1.99M ShadeNet-6 XS to 0.136 after seven epochs). We did not ablate them one by one, so we cannot say how much each contributes.
  • Most of the gain came in the first epochs: phase 1 fell from 0.165 to 0.143 in three epochs, then moved by less than the validation noise (about ±0.004) between epochs 4 and 7.
  • The refiner is nearly redundant here (gain 0.0001 to 0.0017); the main model's output is already smooth.
  • Test-time averaging is not worth it (see Results), and a refiner distilled from it learns little: the teacher is only 0.0038 better than the single pass.
  • The PatchGAN's effect is not isolated. Phase 2 also lowered the learning rate and restarted the optimiser, so the drop from 0.1364 to 0.1243 cannot be credited to the GAN alone, and L1 does not reward sharper detail.

Usage

The inference code is in the inference/ folder of this repository and needs no training code: a single file that rebuilds the network from the checkpoint (verified to give bit-identical outputs to the training code).

pip install torch transformers pillow numpy
python inference/run.py photos/ --checkpoint checkpoints/shadenet6-l.ckpt --out out/
# with the separate refiner (and, for the best recipe measured, the mirror average):
python inference/run.py photos/ --checkpoint checkpoints/shadenet6-l.ckpt --refine --mirror --out out/

Each photo becomes a panel image: input, albedo, depth, normal, at the photo's own shape (the long side is scaled to 448, the rest of the square is filled by mirroring the photo's edge, and the padding is cut off the maps; --framing crop takes the centre square, which is what the model was trained on, and --mirror averages the prediction with the mirrored-photo prediction). The whole-photo framing was not evaluated for this model: the numbers above use square crops.

From Python:

import torch
from inference.shadenet6 import ShadeNet6

model = ShadeNet6.from_checkpoint("checkpoints/shadenet6-l.ckpt", device="cuda")
photo = ...                      # float32 [1, 3, 448, 448] in [-1, 1]
maps = model.predict(photo)      # [1, 7, 448, 448] in [-1, 1]
albedo, depth, normal = maps[:, 0:3], maps[:, 3:4], maps[:, 4:7]

The backbone loads its configuration from facebook/dinov2-base on the Hub (a few kB; the weights come from the checkpoint). The checkpoint is 394 MB (fp32, inference only). config.json lists the architecture, the input and output convention and the validation numbers.

With the int8 ONNX file instead (no PyTorch model, no transformers; see Quantised ONNX for what it is and is not good for):

pip install onnxruntime torch pillow numpy
python inference/run.py photos/ --checkpoint onnx/shadenet6_l_int8.onnx --out out/
import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("onnx/shadenet6_l_int8.onnx")
# photo: float32 [1, 3, 448, 448] in [-1, 1]
maps = sess.run(None, {"input_rgb": photo})[0]   # [1, 7, 448, 448], normals already unit length
albedo, depth, normal = maps[:, 0:3], maps[:, 3:4], maps[:, 4:7]

Limitations

  • Tiny validation set (23 images): the score moves by 0.003 or more between epochs, one run is not a distribution, and the checkpoint was chosen on the same 23 images, so the reported number is optimistic by roughly 0.001 to 0.003.
  • Pseudo-labels: the labels are model predictions (Marigold-V2), not measurements, so the model inherits their errors, and a model scored against them cannot be shown to be better than the labelling model.
  • Size: 98.4M parameters (86.6M of them a frozen backbone) against ShadeNet-5's 34.65M, so this is a comparison of quality at a larger size, not at equal size.
  • Relative depth only: no metric scale.
  • Fixed input sizes: height and width must be multiples of 14 and 32 (448, 896, ...).
  • Whole-photo framing and test-time averaging beyond the mirror were not evaluated for this model. The separate refiner was trained on 448 px centre crops of Open Images photos and scored only on the 23 validation images.
  • Quantisation: an int8 file (lossless) and an int4 file (lossy) are provided; both save size, not time (see Quantised ONNX). GPU speed was measured on one GPU, CPU speed on a shared cloud CPU slice.

License & attribution

Code and weights: Apache-2.0.

  • Backbone: DINOv2 ViT-B/14 (facebook/dinov2-base, Meta AI, Apache-2.0), kept frozen; its weights are part of the checkpoint.
  • Labels: the albedo, normal and depth training labels are pseudo-labels generated with Marigold V2 (huawei-bayerlab/marigold-v2-0, Apache-2.0; its base model Qwen-Image-Edit-2509 is Apache-2.0 too). They are model outputs, not ground truth.
  • Refiner training photos: 8,016 photos from the Open Images V7 validation split that are listed as CC BY 2.0 (annotations CC BY 4.0). Authors and sources are in refiner/attribution.csv; the photos are not redistributed, only the 0.9M-parameter refiner trained on them.
  • Images: the photographs come from the Flickr8k corpus (Hodosh, Young and Hockenmaier, 2013; Rashtchian et al., 2010). The corpus authors release the captions under CC BY-SA, but the photographs are individual Flickr images owned by their photographers and have no blanket licence. The example images on this card are shown for illustration.
Downloads last month
37
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for singam96/ShadeNet-6-L-98M

Quantized
(10)
this model