Model card and the original license

README.md

@@ -0,0 +1,109 @@
+---
+base_model: hustvl/Moebius
+base_model_relation: quantized
+license: mit
+pipeline_tag: image-to-image
+library_name: onnx
+tags:
+- onnx
+- moonbox
+- moebius
+- inpainting
+- object-removal
+- image-to-image
+- diffusion
+---
+
+# Moebius ONNX
+
+ONNX builds of [Moebius](https://huggingface.co/hustvl/Moebius), made by Moonbox for on-device use with ONNX Runtime (CUDA, DirectML, CPU).
+Each precision is a folder with one graph per part of the pipeline; large weights sit next to each graph as ONNX
+external data (`*.onnx.data`), so keep a folder's files together.
+
+## Files
+
+The last column is how far each graph's output is from the original PyTorch model run in fp32 on the same inputs
+(relative RMS error). fp32 is the yardstick because the original's own bf16 run is not exact either. Image-quality
+scores are added as each precision passes the image eval.
+
+| Precision | File | Size | sha256 | Checked against fp32 |
+|---|---|---|---|---|
+| fp16 | [`fp16/unet/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/unet/model.onnx) | 0.91 GB | `3ab9796281da` | fp32 graph (fp16 compute: 10 % off) |
+| fp16 | [`fp16/vae_decoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/vae_decoder/model.onnx) | 0.10 GB | `2818a7cbd433` | 1.9e-04 |
+| fp16 | [`fp16/vae_encoder/model.onnx`](https://huggingface.co/moonbox-dev/Moebius-ONNX/blob/main/fp16/vae_encoder/model.onnx) | 0.07 GB | `8e37212ea885` | 6.3e-05 |
+
+## How to run
+
+Moebius (0.22 B) is an image-inpainting / object-removal model: give it a picture and a mask (white = remove /
+repaint) and it fills the masked region from the surroundings. No text prompt. These folders hold its three graphs
+at 512 px (the short side the original pipeline uses): `vae_encoder`, `unet` (with classifier-free guidance and the
+learned prompt embeddings inside) and `vae_decoder`. The VAE is PixelHacker's (the SDXL VAE, scaling 0.13025); it
+stays fp32 where fp16 overflows (see the table).
+
+**Run it** (ONNX Runtime + diffusers' DDIM scheduler; the original's defaults: 20 steps at strength 0.99, CFG 2.5,
+noise offset 0.0357, result pasted back under a blurred mask):
+
+```python
+import numpy as np, onnxruntime as ort, torch
+from diffusers import DDIMScheduler
+from PIL import Image, ImageFilter
+
+d, prov = "int8", ["CUDAExecutionProvider", "CPUExecutionProvider"]
+enc, unet, dec = (ort.InferenceSession(f"{d}/{p}/model.onnx", providers=prov) for p in ("vae_encoder", "unet", "vae_decoder"))
+img = Image.open("photo.jpg").convert("RGB").resize((512, 512), Image.LANCZOS)    # graphs are 512 x 512
+mask = Image.open("mask.png").convert("L").resize((512, 512), Image.NEAREST)
+x = np.asarray(img, np.float32).transpose(2, 0, 1)[None] / 127.5 - 1
+m = (np.asarray(mask, np.float32)[None, None] / 255 >= 0.5).astype(np.float32)
+lat, mlat = enc.run(None, {"image": x})[0], enc.run(None, {"image": x * (1 - m)})[0]
+m64 = m[:, :, ::8, ::8].copy()
+s = DDIMScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", clip_sample=False)
+s.set_timesteps(20); ts = s.timesteps[1:]                                          # strength 0.99: 19 steps
+g = torch.Generator().manual_seed(0)
+noise = torch.randn(lat.shape, generator=g) + 0.0357 * torch.randn((1, 4, 1, 1), generator=g)
+z = s.add_noise(torch.from_numpy(lat), noise, ts[:1])
+for t in ts:
+    eps = unet.run(None, {"latent": z.numpy(), "mask": m64, "masked_latent": mlat,
+                          "timestep": np.array([float(t)], np.float32), "guidance": np.array([2.5], np.float32)})[0]
+    z = s.step(torch.from_numpy(eps), t, z).prev_sample
+y = dec.run(None, {"latent": (z.numpy() / 0.13025).astype(np.float32)})[0][0]
+out = Image.fromarray((np.clip((y.transpose(1, 2, 0) + 1) / 2, 0, 1) * 255).astype(np.uint8))
+blur = np.asarray(mask.convert("RGB").filter(ImageFilter.GaussianBlur(3)), np.float32) / 255
+Image.fromarray((np.asarray(out) * blur + np.asarray(img) * (1 - blur)).astype(np.uint8)).save("filled.png")
+```
+
+**Precisions.** fp16: every graph in fp16 where it stays within 5 % of fp32 on a test input, otherwise fp32 (the
+table says which). int8: the UNet's weights int8 per output channel (weight-only), VAE graphs as in fp16. int4: the
+UNet's MatMul weights 4-bit (MatMulNBits, block 32) and its other weights int8, VAE graphs as in fp16. No bf16 (no
+bf16 convolution in ONNX Runtime's CUDA provider).
+
+**Benchmark.** BSD100's 100 photos with seeded free-form masks (10-26 % of the picture), the pipeline above on an RTX
+3080 Laptop GPU with ONNX Runtime 1.30: PSNR / SSIM / LPIPS against the original photo and PSNR against the fp32
+pipeline's result (same seed). Inpainting has no single right answer, so the ground-truth numbers are for comparing
+precisions, not a quality score.
+
+| Pipeline | Size | vs fp32 (PSNR, same seed) | PSNR / SSIM / LPIPS vs original | Time / picture | Peak VRAM |
+|---|---|---|---|---|---|
+| fp32 (reference, not published) | 1.24 GB | - | 26.35 / 0.906 / 0.067 | 3.8 s | 7.8 GB |
+| fp16 (UNet fp32) | 1.07 GB | 46.5 dB (worst 35.6) | 26.35 / 0.906 / 0.067 | 3.6 s | 5.9 GB |
+| int8 | 0.44 GB | 40.3 dB (worst 32.5) | 26.38 / 0.906 / 0.066 | 3.9 s | 5.9 GB |
+
+## How these were made
+
+From [Moebius](https://huggingface.co/hustvl/Moebius) at revision `cd01f47fb6`: each part exported with `torch.onnx`, then converted or quantized
+with ONNX Runtime's tools (fp16 / bf16 casts, int8 and int4 weight quantization). Every graph was checked against
+the original before upload; the `sha256` column above is what the files must hash to.
+
+## License
+
+MIT (weights; the GitHub code is Apache-2.0). These files are a derivative of the original model and carry its license: it is copied in this
+repository, and it governs your use of these files. The original model card follows unchanged.
+
+---
+
+# Original model card
+
+《Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance》  
+Project Page: https://hustvl.github.io/Moebius  
+Paper: https://arxiv.org/abs/2606.19195  
+Code: https://github.com/hustvl/Moebius  
+weight: https://huggingface.co/hustvl/Moebius/tree/main