Model card and the original license

6bcc309 verified 2 days ago

6.6 kB

---
base_model: hustvl/Moebius
base_model_relation: quantized
license: mit
pipeline_tag: image-to-image
library_name: onnx
tags:
- onnx
- moonbox
- moebius
- inpainting
- object-removal
- image-to-image
- diffusion
---
# Moebius ONNX

ONNX builds of [Moebius](https://huggingface.co/hustvl/Moebius), made by Moonbox for on-device use with ONNX Runtime (CUDA, DirectML, CPU).
Each precision is a folder with one graph per part of the pipeline; large weights sit next to each graph as ONNX
external data (`*.onnx.data`), so keep a folder's files together.

## Files

The last column is how far each graph's output is from the original PyTorch model run in fp32 on the same inputs
(relative RMS error). fp32 is the yardstick because the original's own bf16 run is not exact either. Image-quality
scores are added as each precision passes the image eval.

| Precision | File | Size | sha256 | Checked against fp32 |
|---|---|---|---|---|
| fp16 | [`fp16/unet/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/unet/model.onnx) | 0.91 GB | `3ab9796281da` | fp32 graph (fp16 compute: 10 % off) |
| fp16 | [`fp16/vae_decoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/vae_decoder/model.onnx) | 0.10 GB | `2818a7cbd433` | 1.9e-04 |
| fp16 | [`fp16/vae_encoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/vae_encoder/model.onnx) | 0.07 GB | `8e37212ea885` | 6.3e-05 |
| int8 | [`int8/unet/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/int8/unet/model.onnx) | 0.27 GB | `6bed99be95b9` | 2.3 % (int8 weights) |
| int8 | [`int8/vae_decoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/int8/vae_decoder/model.onnx) | 0.10 GB | `2818a7cbd433` | 1.9e-04 (fp16 graph) |
| int8 | [`int8/vae_encoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/int8/vae_encoder/model.onnx) | 0.07 GB | `8e37212ea885` | 6.3e-05 (fp16 graph) |

## How to run

Moebius (0.22 B) is an image-inpainting / object-removal model: give it a picture and a mask (white = remove /
repaint) and it fills the masked region from the surroundings. No text prompt. These folders hold its three graphs
at 512 px (the short side the original pipeline uses): `vae_encoder`, `unet` (with classifier-free guidance and the
learned prompt embeddings inside) and `vae_decoder`. The VAE is PixelHacker's (the SDXL VAE, scaling 0.13025); it
stays fp32 where fp16 overflows (see the table).

**Run it** (ONNX Runtime + diffusers' DDIM scheduler; the original's defaults: 20 steps at strength 0.99, CFG 2.5,
noise offset 0.0357, result pasted back under a blurred mask):

```python
import numpy as np, onnxruntime as ort, torch
from diffusers import DDIMScheduler
from PIL import Image, ImageFilter
d, prov = "int8", ["CUDAExecutionProvider", "CPUExecutionProvider"]
enc, unet, dec = (ort.InferenceSession(f"{d}/{p}/model.onnx", providers=prov) for p in ("vae_encoder", "unet", "vae_decoder"))
img = Image.open("photo.jpg").convert("RGB").resize((512, 512), Image.LANCZOS)    # graphs are 512 x 512
mask = Image.open("mask.png").convert("L").resize((512, 512), Image.NEAREST)
x = np.asarray(img, np.float32).transpose(2, 0, 1)[None] / 127.5 - 1
m = (np.asarray(mask, np.float32)[None, None] / 255 >= 0.5).astype(np.float32)
lat, mlat = enc.run(None, {"image": x})[0], enc.run(None, {"image": x * (1 - m)})[0]
m64 = m[:, :, ::8, ::8].copy()
s = DDIMScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", clip_sample=False)
s.set_timesteps(20); ts = s.timesteps[1:]                                          # strength 0.99: 19 steps
g = torch.Generator().manual_seed(0)
noise = torch.randn(lat.shape, generator=g) + 0.0357 * torch.randn((1, 4, 1, 1), generator=g)
z = s.add_noise(torch.from_numpy(lat), noise, ts[:1])
for t in ts:
    eps = unet.run(None, {"latent": z.numpy(), "mask": m64, "masked_latent": mlat,
                          "timestep": np.array([float(t)], np.float32), "guidance": np.array([2.5], np.float32)})[0]
    z = s.step(torch.from_numpy(eps), t, z).prev_sample
y = dec.run(None, {"latent": (z.numpy() / 0.13025).astype(np.float32)})[0][0]
out = Image.fromarray((np.clip((y.transpose(1, 2, 0) + 1) / 2, 0, 1) * 255).astype(np.uint8))
blur = np.asarray(mask.convert("RGB").filter(ImageFilter.GaussianBlur(3)), np.float32) / 255
Image.fromarray((np.asarray(out) * blur + np.asarray(img) * (1 - blur)).astype(np.uint8)).save("filled.png")
```

**Precisions.** fp16: every graph in fp16 where it stays within 5 % of fp32 on a test input, otherwise fp32 (the
table says which). int8: the UNet's weights int8 per output channel (weight-only), VAE graphs as in fp16. int4: the
UNet's MatMul weights 4-bit (MatMulNBits, block 32) and its other weights int8, VAE graphs as in fp16. No bf16 (no
bf16 convolution in ONNX Runtime's CUDA provider).

**Benchmark.** BSD100's 100 photos with seeded free-form masks (10-26 % of the picture), the pipeline above on an RTX
3080 Laptop GPU with ONNX Runtime 1.30: PSNR / SSIM / LPIPS against the original photo and PSNR against the fp32
pipeline's result (same seed). Inpainting has no single right answer, so the ground-truth numbers are for comparing
precisions, not a quality score.

| Pipeline | Size | vs fp32 (PSNR, same seed) | PSNR / SSIM / LPIPS vs original | Time / picture | Peak VRAM |
|---|---|---|---|---|---|
| fp32 (reference, not published) | 1.24 GB | - | 26.35 / 0.906 / 0.067 | 3.8 s | 7.8 GB |
| fp16 (UNet fp32) | 1.07 GB | 46.5 dB (worst 35.6) | 26.35 / 0.906 / 0.067 | 3.6 s | 5.9 GB |
| int8 | 0.44 GB | 40.3 dB (worst 32.5) | 26.38 / 0.906 / 0.066 | 3.9 s | 5.9 GB |

## How these were made

From [Moebius](https://huggingface.co/hustvl/Moebius) at revision `cd01f47fb6`: each part exported with `torch.onnx`, then converted or quantized
with ONNX Runtime's tools (fp16 / bf16 casts, int8 and int4 weight quantization). Every graph was checked against
the original before upload; the `sha256` column above is what the files must hash to.

## License

MIT (weights; the GitHub code is Apache-2.0). These files are a derivative of the original model and carry its license: it is copied in this
repository, and it governs your use of these files. The original model card follows unchanged.

---

# Original model card

《Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance》  
Project Page: https://hustvl.github.io/Moebius  
Paper: https://arxiv.org/abs/2606.19195  
Code: https://github.com/hustvl/Moebius  
weight: https://huggingface.co/hustvl/Moebius/tree/main

	---
	base_model: hustvl/Moebius
	base_model_relation: quantized
	license: mit
	pipeline_tag: image-to-image
	library_name: onnx
	tags:
	- onnx
	- moonbox
	- moebius
	- inpainting
	- object-removal
	- image-to-image
	- diffusion
	---

	# Moebius ONNX

	ONNX builds of [Moebius](https://huggingface.co/hustvl/Moebius), made by Moonbox for on-device use with ONNX Runtime (CUDA, DirectML, CPU).
	Each precision is a folder with one graph per part of the pipeline; large weights sit next to each graph as ONNX
	external data (`*.onnx.data`), so keep a folder's files together.

	## Files

	The last column is how far each graph's output is from the original PyTorch model run in fp32 on the same inputs
	(relative RMS error). fp32 is the yardstick because the original's own bf16 run is not exact either. Image-quality
	scores are added as each precision passes the image eval.

	\| Precision \| File \| Size \| sha256 \| Checked against fp32 \|
	\|---\|---\|---\|---\|---\|
	\| fp16 \| [`fp16/unet/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/unet/model.onnx) \| 0.91 GB \| `3ab9796281da` \| fp32 graph (fp16 compute: 10 % off) \|
	\| fp16 \| [`fp16/vae_decoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/vae_decoder/model.onnx) \| 0.10 GB \| `2818a7cbd433` \| 1.9e-04 \|
	\| fp16 \| [`fp16/vae_encoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/vae_encoder/model.onnx) \| 0.07 GB \| `8e37212ea885` \| 6.3e-05 \|
	\| int8 \| [`int8/unet/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/int8/unet/model.onnx) \| 0.27 GB \| `6bed99be95b9` \| 2.3 % (int8 weights) \|
	\| int8 \| [`int8/vae_decoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/int8/vae_decoder/model.onnx) \| 0.10 GB \| `2818a7cbd433` \| 1.9e-04 (fp16 graph) \|
	\| int8 \| [`int8/vae_encoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/int8/vae_encoder/model.onnx) \| 0.07 GB \| `8e37212ea885` \| 6.3e-05 (fp16 graph) \|

	## How to run

	Moebius (0.22 B) is an image-inpainting / object-removal model: give it a picture and a mask (white = remove /
	repaint) and it fills the masked region from the surroundings. No text prompt. These folders hold its three graphs
	at 512 px (the short side the original pipeline uses): `vae_encoder`, `unet` (with classifier-free guidance and the
	learned prompt embeddings inside) and `vae_decoder`. The VAE is PixelHacker's (the SDXL VAE, scaling 0.13025); it
	stays fp32 where fp16 overflows (see the table).

	Run it (ONNX Runtime + diffusers' DDIM scheduler; the original's defaults: 20 steps at strength 0.99, CFG 2.5,
	noise offset 0.0357, result pasted back under a blurred mask):

	```python
	import numpy as np, onnxruntime as ort, torch
	from diffusers import DDIMScheduler
	from PIL import Image, ImageFilter

	d, prov = "int8", ["CUDAExecutionProvider", "CPUExecutionProvider"]
	enc, unet, dec = (ort.InferenceSession(f"{d}/{p}/model.onnx", providers=prov) for p in ("vae_encoder", "unet", "vae_decoder"))
	img = Image.open("photo.jpg").convert("RGB").resize((512, 512), Image.LANCZOS) # graphs are 512 x 512
	mask = Image.open("mask.png").convert("L").resize((512, 512), Image.NEAREST)
	x = np.asarray(img, np.float32).transpose(2, 0, 1)[None] / 127.5 - 1
	m = (np.asarray(mask, np.float32)[None, None] / 255 >= 0.5).astype(np.float32)
	lat, mlat = enc.run(None, {"image": x})[0], enc.run(None, {"image": x * (1 - m)})[0]
	m64 = m[:, :, ::8, ::8].copy()
	s = DDIMScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", clip_sample=False)
	s.set_timesteps(20); ts = s.timesteps[1:] # strength 0.99: 19 steps
	g = torch.Generator().manual_seed(0)
	noise = torch.randn(lat.shape, generator=g) + 0.0357 * torch.randn((1, 4, 1, 1), generator=g)
	z = s.add_noise(torch.from_numpy(lat), noise, ts[:1])
	for t in ts:
	eps = unet.run(None, {"latent": z.numpy(), "mask": m64, "masked_latent": mlat,
	"timestep": np.array([float(t)], np.float32), "guidance": np.array([2.5], np.float32)})[0]
	z = s.step(torch.from_numpy(eps), t, z).prev_sample
	y = dec.run(None, {"latent": (z.numpy() / 0.13025).astype(np.float32)})[0][0]
	out = Image.fromarray((np.clip((y.transpose(1, 2, 0) + 1) / 2, 0, 1) * 255).astype(np.uint8))
	blur = np.asarray(mask.convert("RGB").filter(ImageFilter.GaussianBlur(3)), np.float32) / 255
	Image.fromarray((np.asarray(out) * blur + np.asarray(img) * (1 - blur)).astype(np.uint8)).save("filled.png")
	```

	Precisions. fp16: every graph in fp16 where it stays within 5 % of fp32 on a test input, otherwise fp32 (the
	table says which). int8: the UNet's weights int8 per output channel (weight-only), VAE graphs as in fp16. int4: the
	UNet's MatMul weights 4-bit (MatMulNBits, block 32) and its other weights int8, VAE graphs as in fp16. No bf16 (no
	bf16 convolution in ONNX Runtime's CUDA provider).

	Benchmark. BSD100's 100 photos with seeded free-form masks (10-26 % of the picture), the pipeline above on an RTX
	3080 Laptop GPU with ONNX Runtime 1.30: PSNR / SSIM / LPIPS against the original photo and PSNR against the fp32
	pipeline's result (same seed). Inpainting has no single right answer, so the ground-truth numbers are for comparing
	precisions, not a quality score.

	\| Pipeline \| Size \| vs fp32 (PSNR, same seed) \| PSNR / SSIM / LPIPS vs original \| Time / picture \| Peak VRAM \|
	\|---\|---\|---\|---\|---\|---\|
	\| fp32 (reference, not published) \| 1.24 GB \| - \| 26.35 / 0.906 / 0.067 \| 3.8 s \| 7.8 GB \|
	\| fp16 (UNet fp32) \| 1.07 GB \| 46.5 dB (worst 35.6) \| 26.35 / 0.906 / 0.067 \| 3.6 s \| 5.9 GB \|
	\| int8 \| 0.44 GB \| 40.3 dB (worst 32.5) \| 26.38 / 0.906 / 0.066 \| 3.9 s \| 5.9 GB \|

	## How these were made

	From [Moebius](https://huggingface.co/hustvl/Moebius) at revision `cd01f47fb6`: each part exported with `torch.onnx`, then converted or quantized
	with ONNX Runtime's tools (fp16 / bf16 casts, int8 and int4 weight quantization). Every graph was checked against
	the original before upload; the `sha256` column above is what the files must hash to.

	## License

	MIT (weights; the GitHub code is Apache-2.0). These files are a derivative of the original model and carry its license: it is copied in this
	repository, and it governs your use of these files. The original model card follows unchanged.

	---

	# Original model card

	《Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance》
	Project Page: https://hustvl.github.io/Moebius
	Paper: https://arxiv.org/abs/2606.19195
	Code: https://github.com/hustvl/Moebius
	weight: https://huggingface.co/hustvl/Moebius/tree/main