--- base_model: ZhengPeng7/BiRefNet_lite base_model_relation: quantized license: mit pipeline_tag: image-segmentation library_name: onnx tags: - onnx - moonbox - birefnet - background-removal - image-segmentation - dichotomous-image-segmentation --- # BiRefNet-lite ONNX ONNX builds of [BiRefNet-lite](https://huggingface.co/ZhengPeng7/BiRefNet_lite), made by Moonbox for on-device use with ONNX Runtime (CUDA, DirectML, CPU). The model is a single graph: each precision is one folder holding `model.onnx`. ## Files The last column is the benchmark: each precision's output against the fp32 graph's on the same eval pictures, and its own score on the eval set (details under *How to run*). | Precision | File | Size | sha256 | Benchmark | |---|---|---|---|---| | fp16 | [`fp16/model.onnx`](https://huggingface.co/moonbox-dev/BiRefNet_lite-ONNX/blob/main/fp16/model.onnx) | 0.11 GB | `311cfd8088ee` | vs fp32: IoU 0.9981, MAE 0.0002; DIS5K MAE 0.0171, S 0.939 | | int8 | [`int8/model.onnx`](https://huggingface.co/moonbox-dev/BiRefNet_lite-ONNX/blob/main/int8/model.onnx) | 0.09 GB | `97dd832a68fe` | vs fp32: IoU 0.9972, MAE 0.0004; DIS5K MAE 0.0171, S 0.939 | | int4 | [`int4/model.onnx`](https://huggingface.co/moonbox-dev/BiRefNet_lite-ONNX/blob/main/int4/model.onnx) | 0.08 GB | `f1569d762955` | vs fp32: IoU 0.9762, MAE 0.0034; DIS5K MAE 0.0172, S 0.939 | ## How to run BiRefNet_lite is the light general-purpose BiRefNet (Swin-T backbone), about 4x smaller than BiRefNet. Made from the official fp32 ONNX `BiRefNet-general-bb_swin_v1_tiny-epoch_232.onnx` from [BiRefNet's GitHub release v1](https://github.com/ZhengPeng7/BiRefNet/releases/tag/v1) (Swin-T backbone); its single output is the matte (sigmoid applied). **ONNX Runtime** (1024 x 1024 input, RGB in 0-1, ImageNet-normalized): ```python import numpy as np, onnxruntime as ort from PIL import Image sess = ort.InferenceSession("int8/model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"]) img = Image.open("photo.jpg").convert("RGB") x = np.asarray(img.resize((1024, 1024), Image.BILINEAR), np.float32) / 255 x = (x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225] # ImageNet normalization x = x.transpose(2, 0, 1)[None] matte = sess.run(None, {sess.get_inputs()[0].name: x})[0][0, 0] alpha = Image.fromarray((np.clip(matte, 0, 1) * 255).astype(np.uint8)).resize(img.size, Image.BILINEAR) img.putalpha(alpha); img.save("cutout.png") ``` **Precisions.** fp16: weights and math in fp16, inputs and outputs fp32. int8: weights int8 per output channel behind DequantizeLinear, activations in float (weight-only: no calibration). int4: MatMul weights 4-bit (MatMulNBits, block 32), the remaining convolution weights int8 weight-only. bf16 is not made: ONNX Runtime's CUDA provider has no bf16 convolution. **Benchmark.** [DIS5K](https://github.com/xuebinqin/DIS) test sets, 250 pictures (DIS-TE1..4 and DIS-VD, 50 each), ONNX Runtime 1.30 CUDA provider on an RTX 3080 Laptop GPU. The table's last column gives the precision's agreement with the fp32 graph (IoU of the masks at 0.5, mean absolute difference) and its own MAE against the ground truth. Scored with PySODMetrics on masks shrunk to at most 2048 px on the long side (DIS5K's largest are about 6000 x 4000); HCE (human correction effort) is given for the reference precision. | Precision | Size | vs fp32 (mask IoU, MAE) | MAE | S-measure | max F | Time / picture | Peak VRAM | |---|---|---|---|---|---|---|---| | fp32 (reference, not published) | 224 MB | - | 0.0171 | 0.939 | 0.945 | 438 ms | 6.7 GB | | fp16 | 114 MB | IoU 0.9981, MAE 0.0002 | 0.0171 | 0.939 | 0.945 | 372 ms | 5.8 GB | | int8 | 92 MB | IoU 0.9972, MAE 0.0004 | 0.0171 | 0.939 | 0.945 | 420 ms | 6.4 GB | | int4 | 82 MB | IoU 0.9762, MAE 0.0034 | 0.0172 | 0.939 | 0.944 | 422 ms | 6.5 GB | ## How these were made From [BiRefNet-lite](https://huggingface.co/ZhengPeng7/BiRefNet_lite) at revision `aa62cd87ea`, starting from an fp32 ONNX graph (where it came from is said above): fp16 with onnxconverter-common (inputs and outputs kept fp32), int8 with ONNX Runtime static QDQ quantization (per-channel int8 weights, uint8 activations, MinMax calibration on pictures outside the eval set), int4 with ONNX Runtime's MatMulNBits (weight-only, block 32) where the graph is mostly matrix multiplications. bf16 is left out for convolutional graphs: ONNX Runtime's CUDA provider has no bf16 convolution. Every graph was benchmarked before upload; the `sha256` column above is what the files must hash to. ## License MIT. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged. --- # Original model card

Bilateral Reference for High-Resolution Dichotomous Image Segmentation

Peng Zheng 1,4,5,6,  Dehong Gao 2,  Deng-Ping Fan 1*,  Li Liu 3,  Jorma Laaksonen 4,  Wanli Ouyang 5,  Nicu Sebe 6
1 Nankai University  2 Northwestern Polytechnical University  3 National University of Defense Technology  4 Aalto University  5 Shanghai AI Laboratory  6 University of Trento 
                 
| *DIS-Sample_1* | *DIS-Sample_2* | | :------------------------------: | :-------------------------------: | | | | This repo is the official implementation of "[**Bilateral Reference for High-Resolution Dichotomous Image Segmentation**](https://arxiv.org/pdf/2401.03407.pdf)" (___CAAI AIR 2024___). Visit our GitHub repo: [https://github.com/ZhengPeng7/BiRefNet](https://github.com/ZhengPeng7/BiRefNet) for more details -- **codes**, **docs**, and **model zoo**! ## How to use (this tiny version) ### 0. Install Packages: ``` pip install -qr https://raw.githubusercontent.com/ZhengPeng7/BiRefNet/main/requirements.txt ``` ### 1. Load BiRefNet: #### Use codes + weights from HuggingFace > Only use the weights on HuggingFace -- Pro: No need to download BiRefNet codes manually; Con: Codes on HuggingFace might not be latest version (I'll try to keep them always latest). ```python # Load BiRefNet with weights from transformers import AutoModelForImageSegmentation birefnet = AutoModelForImageSegmentation.from_pretrained('zhengpeng7/BiRefNet_lite', trust_remote_code=True) ``` #### Use codes from GitHub + weights from HuggingFace > Only use the weights on HuggingFace -- Pro: codes are always the latest; Con: Need to clone the BiRefNet repo from my GitHub. ```shell # Download codes git clone https://github.com/ZhengPeng7/BiRefNet.git cd BiRefNet ``` ```python # Use codes locally from models.birefnet import BiRefNet # Load weights from Hugging Face Models ### >>> Remember to set the `bb` in `config.py` as `swin_v1_t` to use this tiny version. <<< ### birefnet = BiRefNet.from_pretrained('zhengpeng7/BiRefNet_lite') ``` #### Use codes from GitHub + weights from local space > Only use the weights and codes both locally. ```python # Use codes and weights locally ### >>> Remember to set the `bb` in `config.py` as `swin_v1_t` to use this tiny version. <<< ### import torch from utils import check_state_dict birefnet = BiRefNet(bb_pretrained=False) state_dict = torch.load(PATH_TO_WEIGHT, map_location='cpu') state_dict = check_state_dict(state_dict) birefnet.load_state_dict(state_dict) ``` #### Use the loaded BiRefNet for inference ```python # Imports from PIL import Image import matplotlib.pyplot as plt import torch from torchvision import transforms from models.birefnet import BiRefNet birefnet = ... # -- BiRefNet should be loaded with codes above, either way. torch.set_float32_matmul_precision(['high', 'highest'][0]) birefnet.to('cuda') birefnet.eval() birefnet.half() def extract_object(birefnet, imagepath): # Data settings image_size = (1024, 1024) transform_image = transforms.Compose([ transforms.Resize(image_size), transforms.ToTensor(), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) ]) image = Image.open(imagepath) input_images = transform_image(image).unsqueeze(0).to('cuda').half() # Prediction with torch.no_grad(): preds = birefnet(input_images)[-1].sigmoid().cpu() pred = preds[0].squeeze() pred_pil = transforms.ToPILImage()(pred) mask = pred_pil.resize(image.size) image.putalpha(mask) return image, mask # Visualization plt.axis("off") plt.imshow(extract_object(birefnet, imagepath='PATH-TO-YOUR_IMAGE.jpg')[0]) plt.show() ``` > This BiRefNet for standard dichotomous image segmentation (DIS) is trained on **DIS-TR** and validated on **DIS-TEs and DIS-VD**. ## This repo holds the official model weights of "[Bilateral Reference for High-Resolution Dichotomous Image Segmentation](https://arxiv.org/pdf/2401.03407)" (_CAAI AIR 2024_). This repo contains the weights of BiRefNet proposed in our paper, which has achieved the SOTA performance on three tasks (DIS, HRSOD, and COD). Go to my GitHub page for BiRefNet codes and the latest updates: https://github.com/ZhengPeng7/BiRefNet :) #### Try our online demos for inference: + Online **Single Image Inference** on Colab: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/14Dqg7oeBkFEtchaHLNpig2BcdkZEogba?usp=drive_link) + **Online Inference with GUI on Hugging Face** with adjustable resolutions: [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/ZhengPeng7/BiRefNet_demo) + **Inference and evaluation** of your given weights: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1MaEiBfJ4xIaZZn0DqKrhydHB8X97hNXl#scrollTo=DJ4meUYjia6S) ## Acknowledgement: + Many thanks to @fal for their generous support on GPU resources for training better BiRefNet models. + Many thanks to @not-lain for his help on the better deployment of our BiRefNet model on HuggingFace. ## Citation ``` @article{zheng2024birefnet, title={Bilateral Reference for High-Resolution Dichotomous Image Segmentation}, author={Zheng, Peng and Gao, Dehong and Fan, Deng-Ping and Liu, Li and Laaksonen, Jorma and Ouyang, Wanli and Sebe, Nicu}, journal={CAAI Artificial Intelligence Research}, volume = {3}, pages = {9150038}, year={2024} } ```