Add RawNIND UtNet2 (Bayer) for AI Denoise, CC BY 4.0

ce76dbf verified 3 days ago

8.7 kB

license: other
license_name: apache-2.0-mit-and-cc-by-4.0
license_link: https://huggingface.co/pch/rawmakase-models/blob/main/NOTICE
library_name: onnx
pipeline_tag: image-segmentation
tags:
  - onnx
  - sam2
  - detr
  - upernet
  - rawnind
  - denoising
  - image-segmentation
  - rawmakase

RAWmakase model files

The models RAWmakase, an open source RAW photo editor, downloads when you first use Select Subject, Select Sky or Select Background in its Masking tool, or AI Denoise in its Detail panel. They run on your computer with ONNX Runtime; no photo is sent anywhere. The app fetches these files at a pinned commit of this repository and checks each against the SHA-256 below before using it.

Nothing here was trained by RAWmakase. The SAM 2 and DETR files are copies of public ONNX exports of Meta's Apache-2.0 models, kept here so the app's downloads do not depend on third-party repositories. The UperNet file is RAWmakase's own ONNX export of an MIT-licensed checkpoint. The RawNIND file is RAWmakase's ONNX conversion of Benoit Brummer's checkpoint, weights unchanged, under CC BY 4.0. See NOTICE for exact provenance, LICENSE for the Apache license, upernet-convnext-base/LICENSE for the MIT one and rawnind-utnet2-bayer/LICENSE for CC BY 4.0.

File	Bytes	SHA-256
`sam2.1-hiera-small/vision_encoder.onnx`	467440	`aacf1f7137bb6fffcf6bf166abcfabe28f57a76059254f3fb611c4a64a208119`
`sam2.1-hiera-small/vision_encoder.onnx_data`	162476288	`260fd1f0a34e72a3dc79a739e563b4facc0ba75504818b433a1f808e66637456`
`sam2.1-hiera-small/prompt_encoder_mask_decoder.onnx`	213114	`079c59b261f723ff5c6a125e69b0170a957b21c58738c28d2b0394ecd0587d7f`
`sam2.1-hiera-small/prompt_encoder_mask_decoder.onnx_data`	20958208	`f9e59a584ab8ced21fa812c211bc01084204db1c9e92a5ef4fb3a49972b4e864`
`detr-resnet-50-panoptic/detr-panoptic-fp16.onnx`	86559030	`afd9f02d864302d690356fd4bfcb2feed2397a1190bf46a7306cb430464d734a`
`upernet-convnext-base/upernet-convnext-base-sky-fp16.onnx`	242162020	`ed75b2ad81ad209949c9580071ed486a72a423037822a49faa4e98991a43b891`
`rawnind-utnet2-bayer/model_bayer.onnx`	31057086	`5111f19a66a9377002e2ff4ca87de8e3303085b33e6dc11746e42ce811bf6cc1`
`rawnind-utnet2-bayer/LICENSE`	18657	`9ba9550ad48438d0836ddab3da480b3b69ffa0aac7b7878b5a0039e7ab429411`
`rawnind-utnet2-bayer/NOTICE`	777	`809b7f8aac9b4bf7134e7543de867b031f194895b280efebe0206e1ddca868e6`
`rawnind-utnet2-bayer/MODEL_CARD.md`	3670	`3c2d3d55e8bb0d6b4d0b1e5e357ce2cb4aa0e5f0dd6383cd6bf46c44f755ddf6`

Segment Anything Model 2.1, Hiera small

What it does in RAWmakase: draws the outline of each person and animal (and of the sky in earlier releases), and answers clicks when you refine a selection.
Made by: Meta AI (Ravi et al., SAM 2: Segment Anything in Images and Videos, 2024). Weights and code are Apache-2.0: facebookresearch/sam2, facebook/sam2.1-hiera-small. Trained on Meta's SA-1B and SA-V datasets.
This export: onnx-community/sam2.1-hiera-small-ONNX at commit a7df49d8de14b9d2e4504d1687b0d568f905fd8d (float32). An image encoder (pixel_values [1,3,1024,1024], ImageNet mean/std, stretched) and a prompt encoder with mask decoder (points, labels, boxes and the three image embeddings in; three candidate 256×256 mask logits, IoU scores and an object score out). Each graph's weights are in the .onnx_data file beside it.

DETR ResNet-50 panoptic

What it does in RAWmakase: finds the people and animals in the photo (and the sky in earlier releases), so the right outlines are chosen.
Made by: Facebook AI Research (Carion et al., End-to-End Object Detection with Transformers, 2020). Weights and code are Apache-2.0: facebookresearch/detr, facebook/detr-resnet-50-panoptic. Trained on COCO 2017 panoptic.
This export: Xenova/detr-resnet-50-panoptic at commit ea24b2d4e0bfae31f0a1299ba3fb892a2df064de, onnx/model_fp16.onnx (half precision weights, float32 inputs). pixel_values [1,3,H,W] (ImageNet mean/std; RAWmakase uses a long edge of 800) and pixel_mask [1,64,64]; logits [1,100,251] (COCO category ids, last class is "no object"; person 1, animals 16–25, sky 187) and pred_masks [1,100,H/4,W/4]. ONNX Runtime's full graph optimisation fails on this file; RAWmakase loads it with basic optimisation.

UperNet ConvNeXt base

What it does in RAWmakase: finds the sky, for Select Sky.
Made by: OpenMMLab's training of UperNet (Xiao et al., 2018) with ConvNeXt (Liu et al., Meta AI, 2022), openmmlab/upernet-convnext-base, whose card says MIT. Trained on ADE20K (MIT CSAIL), whose images are offered for non-commercial research and education.
This export: made by export_upernet_sky.py from that repository at commit 13a9c410ff779f25236a8d24802d2742e4694e9c, which reproduces it byte for byte with the same package versions. Half precision weights, cast to float32 as it loads; the auxiliary head dropped; adaptive pooling written as the same averages so any size works. pixel_values [1,3,H,W] (ImageNet mean/std; RAWmakase uses a long edge of 1024); sky [1,H,W], the probability of ADE20K's sky at each pixel.

RawNIND UtNet2 (Bayer)

What it does in RAWmakase: AI Denoise denoises and demosaics a RAW photo's mosaic in one pass, on your computer.
Made by: Benoit Brummer (UCLouvain), rawnind_jddc at commit e80fb7b14dc440bba38c01f312fd3cc02deb566c; Brummer and De Vleeschouwer, arXiv:2501.08924. Trained on the RawNIND dataset (doi:10.14428/DVN/DEQCIM). The author licenses the weights under GPL-3.0 or CC BY 4.0; RAWmakase takes and redistributes them under CC BY 4.0 (LICENSE, credit in NOTICE). RAWmakase is not endorsed by the authors.
This export: RAWmakase's conversion of checkpoint DenoiserTrainingBayerToProfiledRGB_4ch_2024-02-21-bayer_ms-ssim_mgout_notrans_valeither_-4 (SHA-256 c1f7de3de24cb12f4803ee65e11cfb467e9a5b80c3cb7126746ef026106c15e0) to ONNX with darktable-ai 76513a1e379b4265007bae768825b9a581944ff8; the weights are unchanged. input [1,4,512,512] (packed RGGB planes, 0–1, not white balanced) and output [1,3,1024,1024] (camera RGB at an arbitrary scale). Details and measurements in its model card.

Citation

@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  year={2024}
}
@inproceedings{xiao2018upernet,
  title={Unified Perceptual Parsing for Scene Understanding},
  author={Xiao, Tete and Liu, Yingcheng and Zhou, Bolei and Jiang, Yuning and Sun, Jian},
  booktitle={ECCV},
  year={2018}
}
@inproceedings{liu2022convnext,
  title={A ConvNet for the 2020s},
  author={Liu, Zhuang and Mao, Hanzi and Wu, Chao-Yuan and Feichtenhofer, Christoph and Darrell, Trevor and Xie, Saining},
  booktitle={CVPR},
  year={2022}
}
@article{brummer2025rawnind,
  title={Learning Joint Denoising, Demosaicing, and Compression from the Raw Natural Image Noise Dataset},
  author={Brummer, Benoit and De Vleeschouwer, Christophe},
  journal={arXiv preprint arXiv:2501.08924},
  year={2025}
}
@inproceedings{carion2020detr,
  title={End-to-End Object Detection with Transformers},
  author={Carion, Nicolas and Massa, Francisco and Synnaeve, Gabriel and Usunier, Nicolas and Kirillov, Alexander and Zagoruyko, Sergey},
  booktitle={ECCV},
  year={2020}
}