Cool2009/SHARP-onnx

SHARP (ONNX, fp32 / fp16 / fp8 / int8 / int4)

ONNX export of SHARP — Sharp Monocular View Synthesis in Less Than a Second (Apple, ICLR 2026).

Given a single photograph, the model regresses a metric 3D Gaussian splat representation of the scene (1,179,648 Gaussians) in a single feed-forward pass. The result can then be rendered from nearby viewpoints at high resolution.

• Paper: arXiv:2512.10685
• Upstream code and weights: apple/ml-sharp
• Project page: https://apple.github.io/ml-sharp/

Unofficial conversion. This repository is a format conversion (.pt → .onnx) of the released checkpoint sharp_2572gikvuh.pt, plus reduced-precision derivatives of that conversion. It is not affiliated with or endorsed by Apple. Because it is a format conversion of the Apple Machine Learning Research Model, it counts as a Model Derivative under the Apple license and is distributed for non-commercial research use only. See License.

Which variant should I use?

Short answer: fp16/ unless you are short on disk or memory, then int4/. Skip int8/.

Folder	Weights	Size	CPU run¹	Quality vs fp32²	Verdict
fp32/	float32	2.62 GB	38.9 s	reference	Exact original export. Use only as the reference.
fp16/	float16	1.31 GB	53.7 s	best (means 0.6 %, colors 0.1 %)	Recommended. Half the size, visually indistinguishable.
int4/	4-bit (MatMulNBits)	0.58 GB	38.9 s	good (means 2.6 %, colors 0.2 %)	Best size/quality trade-off. 22 % of fp32 size, no slower.
fp8/	float8 E4M3	0.66 GB	47.1 s	good (means 3.0 %, colors 0.4 %)	Alternative to int4; slightly better colors, slightly worse scales.
int8/	int8 dynamic	0.72 GB	216.8 s	worst (means 27 %, scales 54 %)	Do not use. Bigger than int4, 5.6× slower than fp32, and visibly wrong.

¹ One 1536×1536 forward pass, ONNX Runtime 1.31.1, CPU (16 cores), best of two runs, warm session. GPU timings will differ (fp16/fp8 are the ones that benefit).

² Relative L2 error against the fp32 output of the same image, per output tensor: mean_vectors (geometry) and colors (appearance) matter most for how a render looks; singular_values and quaternions are noisier by nature because many of their components are near zero or are direction-coded. Full numbers are in the table below.

Measured accuracy (relative L2 error vs fp32, same image)

Variant	mean_vectors	singular_values	quaternions	colors	opacities
fp16	6.3e-3	1.1e-2	4.7e-2	1.2e-3	1.3e-2
int4	2.6e-2	8.9e-2	1.2e-1	2.2e-3	2.3e-2
fp8	3.0e-2	1.3e-1	1.2e-1	4.0e-3	5.2e-2
int8	2.7e-1	5.4e-1	3.5e-1	1.3e-2	1.4e-1

Notes on the odd results:

• fp16 is slower than fp32 on CPU. CPUs have no fp16 kernels, so ONNX Runtime casts to fp32 and back around most nodes. On GPU fp16 is the fastest option.
• int8 is worse than int4. int8 is dynamic quantization: activations are quantized at runtime with a scale computed on the fly, which this transformer-style encoder does not tolerate well. int4 only stores the MatMul weights in 4 bits (block-wise, symmetric, block size 32) and keeps activations in fp32, which is both more accurate and much faster here.
• fp8 needs ONNX Runtime ≥ 1.25 for the float8 DequantizeLinear CPU kernel (tested on 1.31.1). On older builds the model fails to load. It also needs opset 19.

Files

Each variant lives in its own folder and consists of exactly two files:

fp32/model.onnx        graph
fp32/model.onnx.data   external weights (must keep this name, next to the .onnx)
fp16/ ...  fp8/ ...  int8/ ...  int4/ ...

Both files of a folder are required — the .onnx alone will not load. Weights are stored externally because every variant exceeds the 2 GB protobuf limit except int4.

Supporting files at the top level:

File	Purpose
run_onnx.py	Minimal CLI: image in, .npz of the Gaussian parameters out.
quantize.py	Rebuilds the fp16/fp8/int8/int4 variants from fp32/.
requirements.txt	Runtime dependencies for run_onnx.py.
LICENSE	Apple Machine Learning Research Model License Agreement.

Model I/O

Name	Direction	Shape	dtype	Notes
image	input	[1, 3, 1536, 1536]	float32	Fixed resolution, NCHW, RGB. All variants take and return float32.
disparity_factor	input	[1]	float32	Metric scale of the reconstruction; 1.0 for a neutral reconstruction.
mean_vectors	output	[1, 1179648, 3]	float32	Gaussian centers, OpenCV convention (x right, y down, z forward).
singular_values	output	[1, 1179648, 3]	float32	Per-axis Gaussian scales.
quaternions	output	[1, 1179648, 4]	float32	Gaussian rotations.
colors	output	[1, 1179648, 3]	float32	Per-Gaussian RGB color.
opacities	output	[1, 1179648]	float32	Activation already applied (sigmoid).

The network outputs Gaussian parameters directly; rendering them into images is done outside the model (e.g. with gsplat, as in the upstream repository, or any 3DGS viewer).

Usage

pip install -r requirements.txt
python run_onnx.py --image photo.jpg --out gaussians.npz --model int4/model.onnx

Download one variant from the Hub (a folder download pulls both files):

huggingface-cli download <your-username>/sharp-onnx --include "fp16/*" --local-dir .

Minimal Python:

import numpy as np
import onnxruntime as ort

sess = ort.InferenceSession(
    "fp16/model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

image = np.zeros((1, 3, 1536, 1536), np.float32)          # your preprocessed photo
disparity_factor = np.ones((1,), np.float32)

means, scales, quats, colors, opacities = sess.run(
    ["mean_vectors", "singular_values", "quaternions", "colors", "opacities"],
    {"image": image, "disparity_factor": disparity_factor},
)

run_onnx.py resizes the input to 1536×1536 and feeds RGB in [0, 1]. Upstream preprocessing may differ in normalization details; use the ml-sharp pipeline if you need bit-exact parity with the published results.

How the variants were made

quantize.py regenerates all of them from fp32/model.onnx:

python quantize.py fp16   # or int8 / int4 / fp8 / all

• fp16 — onnxruntime.transformers.float16.convert_float_to_float16 with keep_io_types=True, so the graph keeps float32 inputs and outputs; Resize scale tensors are cast back to float32.
• fp8 — weights stored as float8 E4M3 with a per-output-channel scale and restated by a DequantizeLinear node (opset 19). 295 weight tensors converted.
• int8 — quantize_dynamic(weight_type=QInt8, per_channel=True).
• int4 — MatMulNBitsQuantizer(block_size=32, symmetric, 4 bits); only the MatMul weights are quantized, Conv weights stay fp32.

The fp32 export itself is the original PyTorch graph converted to ONNX opset 18, renamed from sharp_2572gikvuh_fp32_single_data.onnx; the embedded external-data references were rewritten to model.onnx.data. No weight values were changed in fp32/.

Performance

• fp32 needs 2.6 GB for the weights plus activations and the 1.18M Gaussians (30 MB per output array); the int4 variant needs ~0.6 GB.
• One 1536×1536 forward pass per image. Real-time speed needs a GPU execution provider; CPU inference works and is slow (see the table).
• The quantized variants trade a small amount of accuracy for 2–4.5× less disk and RAM.

License

This repository redistributes the Apple Machine Learning Research Model. It is licensed under the Apple Machine Learning Research Model License Agreement, reproduced verbatim in LICENSE and available upstream at apple/ml-sharp/LICENSE_MODEL.

Key points:

• Research use only. Non-commercial scientific research and academic development only. No commercial products or services.
• Redistribution requires including a copy of the agreement and the attribution notice “Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.”
• Model Derivatives (this ONNX conversion and its quantized variants included) must be identified as such, with the modifications disclosed: the graph was converted from PyTorch to ONNX opset 18, the external-data file was renamed, and the fp16/fp8/int8/int4 folders contain numerically reduced-precision copies of the same weights. The fp32 weights are unchanged.
• Apple's name and logos may not be used to endorse or promote this repository.

The upstream ml-sharp source code is under the Apple Sample Code License; no upstream source code is redistributed here.

Citation

@inproceedings{Sharp2025:arxiv,
  title     = {Sharp Monocular View Synthesis in Less Than a Second},
  author    = {Lars Mescheder and Wei Dong and Shiwei Li and Xuyang Bai and Marcel Santos and
               Peiyun Hu and Bruno Lecouat and Mingmin Zhen and Ama{\"e}l Delaunoy and Tian Fang and
               Yanghai Tsin and Stephan R. Richter and Vladlen Koltun},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2512.10685}
}