ebowwa/moge-2-vits-normal-coreml

MoGe-2 ViT-S Normal — Core ML

Apple Silicon Core ML conversion of Microsoft/MoGe (commit 925b8ed835a7a9cdb7578ba15c658a0afc969030). All variants share the identical op graph (traced at 1x3x512x768, fp16 compute, macOS 15 target); they differ only in weight storage (const dequantize ops). At runtime all computation is fp32.

Variants (branch per variant)

Precision	Branch	Size	Parity (worst median rel-depth err)	Mask IoU	Storage
fp16	fp16	72.4 MB	worst med-rel-depth n/a	IoU n/a	fp16 (baseline, all-fp32 compute)
int8	int8	36.4 MB	worst med-rel-depth 0.0318	IoU 0.9989	int8 blockwise linear_symmetric, bs32
int4	int4	18.6 MB	worst med-rel-depth 0.1664	IoU 0.9971	int4 blockwise linear (asymmetric), bs32
int4-sym	int4-sym	18.5 MB	worst med-rel-depth 0.2104	IoU 0.9974	int4 blockwise linear_symmetric, b64
pal8	pal8	36.4 MB	worst med-rel-depth 0.0336	IoU 0.999	int8 palettized (kmeans, 8-bit)

Parity = per-scene median relative depth error vs the author's reference outputs (03 scenes, same photos/artifacts as the upstream moge infer release test), computed with this repo's harness (--diff). fp16/int8/pal8 match the reference; int4 (asymmetric) and int4-sym carry a small-accuracy-gap documented per scene. The int4 palettized 4-bit variant does NOT meet parity (~0.90) and is excluded. FOV: int8/pal8 within ~0.3 deg of reference.

Per-scene parity vs reference

int8 — int8 blockwise linear_symmetric, bs32

Scene	median rel-depth err	mask IoU	FOV x	FOV y
House indoor	0.0318	1.0000	89.80 deg	67.19 deg
Office	0.0275	1.0000	71.52 deg	51.29 deg
Traffic	0.0252	0.9989	50.28 deg	34.74 deg

int4 — int4 blockwise linear (asymmetric), bs32

Scene	median rel-depth err	mask IoU	FOV x	FOV y
House indoor	0.0388	1.0000	92.94 deg	70.12 deg
Office	0.1664	1.0000	66.86 deg	47.51 deg
Traffic	0.1164	0.9971	55.19 deg	38.42 deg

int4-sym — int4 blockwise linear_symmetric, b64

Scene	median rel-depth err	mask IoU	FOV x	FOV y
House indoor	0.1483	1.0000	90.77 deg	68.09 deg
Office	0.0496	1.0000	71.04 deg	50.89 deg
Traffic	0.2104	0.9974	62.93 deg	44.39 deg

pal8 — int8 palettized (kmeans, 8-bit)

Scene	median rel-depth err	mask IoU	FOV x	FOV y
House indoor	0.0307	1.0000	89.82 deg	67.22 deg
Office	0.0336	1.0000	71.99 deg	51.68 deg
Traffic	0.0201	0.9990	50.27 deg	34.74 deg

Performance (M1 MacBook Air, 16 GB, macOS 26.5.2, CoreML ALL units)

Real-hardware latency: single predict on a 1x3x512x768 input, mean of 5 runs after warmup, measured with the tooling venv (no ANE — this model's ops are arch-gated to CPU/GPU on A13/M1):

Variant	Size	mean latency	FPS
fp16	72.4 MB	112 ms	8.9
int8	36.4 MB	114 ms	8.8
int4	18.6 MB	151 ms	6.6
int4-sym	18.5 MB	155 ms	6.5
pal8	36.4 MB	124 ms	8.1

Provenance

Source repo	Microsoft/MoGe @ 925b8ed835a7a9cdb7578ba15c658a0afc969030
Checkpoint SHA-256	79a16621928c2bf0ed04659218c55c01075e950507f40bb3332fb4c873d3e1dc
Conversion	torch.export -> coremltools 9.0 (macOS15, fp16)
License	MIT (upstream). See LICENSE.md
Date	2026-08-06

Checkpoint SHA-256 verified against the reference RESULTS.md manifest before conversion. This repo is an artifact mirror of an MIT-licensed open model; conversion metadata in provenance.json.

Usage

import coremltools as ct
model = ct.models.MLModel("moge-2-vits-normal-coreml", compute_units=ct.ComputeUnit.ALL)
# input: image (1, 3, 512, 768) float32, RGB, 0..1
out = model.predict({"image": img})
# out["points"] (1,512,768,3) affine point map, out["normal"], out["mask"], out["metric_scale"]

Numerical post-processing (focal/shift recovery, depth map, intrinsics, FOV, UV) is pure numpy — see the accompanying harness/postprocess sources in the publisher's repo (zoo/moge/).

Notes & limitations

• Apple Silicon only (Core ML mlprogram). No iOS deployment target shipped.
• No ANE network on M1/A13 (layer_norm/gelu/sdpa/linear are arch-gated); runs CPU/GPU via CoreML.
• Fixed input size 512x768 — resize before predict.
• int4: recommend for storage-sensitive CPU paths; expect the accuracy gap listed above (blockwise asymmetric linear, bs32).