README.md

---
library_name: mlx
pipeline_tag: image-to-3d
license: other
license_name: sam-license
license_link: LICENSE
base_model:
  - facebook/sam-3d-objects
  - Ruicheng/moge-3-vitl
tags:
  - mlx
  - apple-silicon
  - bfloat16
  - image-to-3d
  - gaussian-splatting
  - inference
---

# SAM 3D Objects — MLX BF16

BF16 weights of Meta SAM 3D Objects converted for the MLX runtime in
[mlx-vlm](https://github.com/Blaizzy/mlx-vlm) (`mlx_vlm/models/sam3d_objects`),
bundled with the MoGe-3 depth estimator
([mlx-community/moge-3-vitl-mlx-fp32](https://huggingface.co/mlx-community/moge-3-vitl-mlx-fp32),
float32) in place of the upstream pipeline's MoGe-v1. This repository holds
weights and metadata only; the inference code lives in mlx-vlm. This is an
unofficial derivative. SAM components retain the SAM License; MoGe-3 retains
MIT; FlexiCubes retains Apache-2.0.

The seven safetensors files hold 3,288 BF16 SAM tensors covering the structure
and pose flow model, the sparse-latent flow model, image and point-map
conditioners, the occupancy decoder, both Gaussian decoders (32 or 4 Gaussians
per voxel), and the mesh decoder, plus the 607 float32 MoGe-3 tensors under
the `depth_model` prefix. The model generates geometry and pose; it has no
text or tool-calling interface.

## Usage

Install an mlx-vlm version that includes `sam3d_objects`, download this
repository, and run the CLI or the Python API:

```sh
pip install mlx-vlm
hf download mlx-community/sam-3d-objects-bf16 --local-dir sam-3d-objects-bf16
python -m mlx_vlm.models.sam3d_objects.generate \
  --model sam-3d-objects-bf16 --image image.png --mask mask.png \
  --output output --formats gaussian mesh
```

```python
from mlx_vlm.models.sam3d_objects.generate import read_image, read_mask
from mlx_vlm.models.sam3d_objects.pipeline import Pipeline

pipeline = Pipeline.from_pretrained("sam-3d-objects-bf16")
image, mask = read_image("image.png")[..., :3], read_mask("mask.png")
result = pipeline.generate(image, mask, seed=42)
```

`pipeline.astream(requests)` streams ordered events for an iterable or async
iterable of `Request` values, one request at a time with backpressure; the
same weights also load through `mlx_vlm.utils.load_model`. Inputs are HWC
RGB/RGBA MLX arrays and HW masks. Automatic MoGe-3 depth is included, in
MoGe-3's metric scale; external HWC point maps use +X left, +Y up, +Z forward
with NaN for invalid points.
Results include local-space Gaussians, mesh vertices/faces/colors, latent
features, occupied voxels, and a separate pose (rotation, translation, scale).
The mlx-vlm model README documents the full API, CLI, JSONL streaming, and
the inference optimizations.

## Conversion and validation

SAM weights were converted directly from the official tensor-only PyTorch
ZIP checkpoints through a restricted stdlib reader. Tensor bytes are first
preserved in safetensors, then mapped to MLX layouts and cast to BF16. The
duplicate occupancy decoder is removed. The MoGe-3 tensors are the
mlx-community MLX conversion, bundled unchanged in float32. Conversion and
the model runtime import neither PyTorch nor NumPy; the CLI reads images with
Pillow and writes PLY/OBJ files with NumPy.

All parameter names/shapes were checked with strict loading, every SAM
floating tensor is BF16, and the loaded model has zero trainable parameters.
Component checks against the original reference in FP32 MLX CPU cover
attention, both flow models, occupancy decoding, interpolation, and
FlexiCubes. Maximum absolute error was below 0.000002 on these fixtures;
mesh face indices matched exactly. The MoGe-3 port is validated against its
PyTorch reference in the `moge3` model repository. This is not a claim of
bitwise BF16 end-to-end parity or a reconstruction-quality evaluation.

See `validation.json` and `conversion.json` for details. The original sources
are [SAM 3D Objects](https://github.com/facebookresearch/sam-3d-objects) and
[MoGe-3](https://github.com/microsoft/MoGe).

### Local performance

Hardware: Apple M5 Max, 128 GiB unified memory. Input: 448×672 RGB plus object mask (id3_shutterstock_WildAnimal_Waterhole_2010559391/image.png and 0.png resized to 672x448);
automatic MoGe-3 depth, 25 structure steps and 25 latent steps, both
Gaussian heads and mesh. One warmup was excluded; the same seed produced
identical latent outputs.

| Metric | Result |
| --- | ---: |
| Median latency (3 runs) | 10.96 s |
| Maximum peak memory | 9.58 GB |

Two additional async requests matched the synchronous latent output while
the event-loop heartbeat continued. These measurements describe one input.