Hugging Face's logo

moonbox-ai
/
efficientvit-MLX

Mask Generation
MLX
Safetensors
moonbox
efficientvit
efficientvit-sam
segment-anything
image-segmentation
Model card Files
xet
 Community
How to use from the
Use from the
MLX library
# Download the model from the Hub
pip install huggingface_hub[hf_xet]

hf download moonbox-ai/efficientvit-MLX --local-dir efficientvit-MLX

EfficientViT-SAM MLX

MLX builds of EfficientViT-SAM, made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's files together. Only the parts listed below are converted: the rest of the pipeline comes from the original repository unless the section below says otherwise.

Files

The last column is how far each build is from the original PyTorch model on the same inputs (see the section below for what was measured).

Precision File Size sha256 Checked against the original
bf16 bf16/config.json 0.3 kB 9e98792690f7 below threshold (experimental): mask IoU 0.714 mean / 0.091 worst
bf16 bf16/mlx_model.py 25.4 kB 42cea84fb312 below threshold (experimental): mask IoU 0.714 mean / 0.091 worst
bf16 bf16/model.safetensors 122.7 MB e7acc4ef2253 below threshold (experimental): mask IoU 0.714 mean / 0.091 worst
fp16 fp16/config.json 0.3 kB 9b286f991620 below threshold (experimental): mask IoU 0.942 mean / 0.611 worst (float32 port: 0.9996 / 0.997)
fp16 fp16/mlx_model.py 25.4 kB 42cea84fb312 below threshold (experimental): mask IoU 0.942 mean / 0.611 worst (float32 port: 0.9996 / 0.997)
fp16 fp16/model.safetensors 122.7 MB f4b4b159c9f7 below threshold (experimental): mask IoU 0.942 mean / 0.611 worst (float32 port: 0.9996 / 0.997)

How to run

EfficientViT-SAM L2 (EfficientViT-L2 image encoder at 512 px + SAM's prompt encoder and mask decoder) ported to MLX. Each folder holds model.safetensors, config.json and mlx_model.py, the port (only mlx, numpy and pillow needed), with EfficientViTSamPredictor's interface:

import numpy as np, mlx_model
from PIL import Image
model = mlx_model.load("fp16")
rgb = np.array(Image.open("in.jpg").convert("RGB"))
emb = mlx_model.embed(model, rgb)
masks, scores, low_res, logits = mlx_model.predict(model, emb, points=[[320, 240]], labels=[1], multimask=True)

Both folders are experimental: this model needs float32. Thirty of its 1x1 convolutions are followed by a BatchNorm with a running variance down to 5.5e-7 (scales up to 600): those channels carry their signal in the 4th-5th significant digit of the convolution output, which 16-bit weights or arithmetic can't hold. The port itself is exact: run in float32 (with MLX_ENABLE_TF32=0, else MLX's float32 matmuls use TF32) it matches PyTorch to mask IoU 0.9996. There are no 8bit / 6bit / 4bit folders: quantized, the model gives no usable masks.

Precision Size Mask IoU vs PyTorch fp32 (mean / worst) Status
fp16/ 122.7 MB 0.942 / 0.611 below threshold (0.95), experimental
bf16/ 122.7 MB 0.714 / 0.091 below threshold, experimental

To run in float32, load either folder and cast the parameters up (model.update(mlx.utils.tree_map(lambda a: a.astype(mx.float32), model.parameters()))): fp16 weights in float32 give 0.966 mean / 0.62 worst, still short of the original's float32 weights. About 55 ms per image (encoder + 2 prompts) on an Apple M5.

How these were made

From EfficientViT-SAM at revision de7d7733cc: the original weights loaded in MLX and stored as bf16 or fp16, or quantized with MLX's affine quantization (8bit, 6bit, 4bit, group size 64; layers MLX can't quantize, such as convolutions and norms, stay bf16). Every build was checked against the original before upload; the sha256 column above is what the files must hash to.

License

apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.

Original model card

Efficient Vision Foundation Models for High-Resolution Generation and Perception

PWC

News

  • (🔥 New) [2025/09/05] We will no longer maintain this codebase. All future updates and announcements will be made on DC-Gen.
  • (🔥 New) [2025/01/24] We released DC-AE-SANA-1.1: doc.
  • (🔥 New) [2025/01/23] DC-AE and SANA are accepted by ICLR 2025.
  • (🔥 New) [2025/01/14] We released DC-AE+USiT models: model, training. Using the default training settings and sampling strategy, DC-AE+USiT-2B achieves 1.72 FID on ImageNet 512x512, surpassing the SOTA diffusion model EDM2-XXL and SOTA auto-regressive image generative models (MAGVIT-v2 and MAR-L).

Content

[ICLR 2025] Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models [paper] [readme] [poster]

Deep Compression Autoencoder (DC-AE) is a new family of high-spatial compression autoencoders with a spatial compression ratio of up to 128 while maintaining reconstruction quality. It accelerates all latent diffusion models regardless of the diffusion model architecture.

Demo

demo

Figure 1: We address the reconstruction accuracy drop of high spatial-compression autoencoders.

demo

Figure 2: DC-AE speeds up latent diffusion models.

Figure 3: DC-AE enables efficient text-to-image generation on the laptop: SANA.

[CVPR 2024 eLVM Workshop] EfficientViT-SAM: Accelerated Segment Anything Model Without Accuracy Loss [paper] [online demo] [readme]

EfficientViT-SAM is a new family of accelerated segment anything models by replacing SAM's heavy image encoder with EfficientViT. It delivers a 48.9x measured TensorRT speedup on A100 GPU over SAM-ViT-H without sacrificing accuracy.

[ICCV 2023] EfficientViT-Classification [paper] [readme]

Efficient image classification models with EfficientViT backbones.

[ICCV 2023] EfficientViT-Segmentation [paper] [readme]

Efficient semantic segmantation models with EfficientViT backbones.

demo

EfficientViT-GazeSAM [readme]

Gaze-prompted image segmentation models capable of running in real time with TensorRT on an NVIDIA RTX 4070.

GazeSAM demo

Getting Started 

conda create -n efficientvit python=3.10
conda activate efficientvit
pip install -U -r requirements.txt

Third-Party Implementation/Integration

Contact

Han Cai

Reference

If EfficientViT or EfficientViT-SAM or DC-AE is useful or relevant to your research, please kindly recognize our contributions by citing our paper:

@inproceedings{cai2023efficientvit,
  title={Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction},
  author={Cai, Han and Li, Junyan and Hu, Muyan and Gan, Chuang and Han, Song},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={17302--17313},
  year={2023}
}

@article{zhang2024efficientvit,
  title={EfficientViT-SAM: Accelerated Segment Anything Model Without Performance Loss},
  author={Zhang, Zhuoyang and Cai, Han and Han, Song},
  journal={arXiv preprint arXiv:2402.05008},
  year={2024}
}

@article{chen2024deep,
  title={Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models},
  author={Chen, Junyu and Cai, Han and Chen, Junsong and Xie, Enze and Yang, Shang and Tang, Haotian and Li, Muyang and Lu, Yao and Han, Song},
  journal={arXiv preprint arXiv:2410.10733},
  year={2024}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
Mask Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Papers for moonbox-ai/efficientvit-MLX