ucalyptus/pixal3d-mlx-m3

Pixal3D on Apple Silicon (MLX), verified on a 16 GB M3

Image → textured PBR mesh (GLB) using Pixal3D, run natively with MLX on an Apple M3 with 16 GB unified memory.

This repository is code, a reproduction recipe, an example output, and measurements. It contains no model weights; they are downloaded from their original sources (see Weights).

It is a fork of lyonsno/pixal3d-mlx at abc9a2e9f18568887dab7cba9814ea101bd0a430. m3-changes.patch is the exact source diff. The full project README is PROJECT_README.md.

What changed from the upstream MLX port

• Export-path fix (the important one). Tencent's exporter reconstructs the predicted surface with narrow-band UDF dual contouring (remesh=True, remesh_band=1, remesh_project=0) and projects texture samples back onto the original surface. The MLX port skipped this and repaired/simplified the raw dual-grid mesh, producing large planar shards and gaps. trellmlx/pixal3d_mesh.py adds the reference-style reconstruction using native Metal CuMesh/MtlBVH backends; texture baking projects onto the pre-remesh surface.
• Objective-C autorelease pools around the native geometry calls, so completed Metal command buffers don't hold several GiB into texture baking.
• Explicit local weight paths (--checkpoint-dir, --dinov3-weights, --naf-weights); resume forwards them plus --quantize.
• Camera-estimation failures and token-budget overruns raise errors instead of silently falling back to a default camera or a lower resolution.

Measured on an Apple M3, 16 GB

Run	Result
512 / INT4 / 1024² atlas, full pipeline (original export)	592.5 s; 11.18 GiB peak MLX allocation; 11.80 GiB peak process footprint; +1.82 GiB swap
Corrected export replay on those cached outputs	43.5 s; 9.14 GiB peak footprint; no added swap
512 / FP16 flows	All geometry stages completed; stopped in texture sampling at the +2 GiB swap safety limit

The full INT4 run needed other apps closed first; with a browser open it hit the swap limit. Treat 512 on 16 GB as workable, with tight headroom, not comfortable.

FP16 vs INT4: at 512, the two meshes are visually indistinguishable (one sample each; flows diverge from the first stage, so this is not a paired numerical comparison).

Against Tencent's hosted CUDA pipeline: the official pipeline supports only 1024 and 1536; 512 is specific to this port. Tencent's 1024 result is sharper (individual lace strands, eyelets, sole tread). Its MoGe camera estimate (0.7078 rad, distance 1.3534) agrees with the native MLX MoGe port (0.7117 rad, 1.3459) to about 0.2°. The upstream MLX port reports ~33 GB peak at 1024, so 1024 does not fit this machine.

Parity checks performed

• Mesh extraction: executing the published upstream function on the cached decoder output gave identical vertices and every triangle index (2,151,698 triangles).
• Sparse convolution (28 asymmetric cases) and subdivision/skip ordering (9 cases) matched CPU Torch exactly; a real decoder block matched FP32 Torch within ~1e-5.
• Native remesher: all 1.6M active cells recomputed on CPU with zero crossing-flag disagreements; 26,161 Metal BVH distances matched an independent CPU closest-triangle search within 1.9e-9, with zero inside/outside disagreements.

Reports: examples/reports/.

Known limitations

• Not watertight. A few hundred incidence-4 (non-manifold) edges remain. They come from checkerboard sign patterns that one-vertex-per-cell dual contouring cannot resolve; the pinned CUDA kernel uses the same rule. Fine for viewing/rendering; not ready for 3D printing or simulation without an ambiguity-resolving remesher.
• Texture conditioning uses 512-pixel NAF features where upstream uses 1024.
• Full CUDA decoder numerical parity is unverified (only mathematical/CPU parity above).
• Tested on one image; general 16 GB reliability is not established.

Setup

uv venv .venv --python python3.12 && source .venv/bin/activate
uv pip install -e .
# Native geometry backends (needed for the export fix):
xcodebuild -downloadComponent MetalToolchain
uv pip install torch==2.12.0 setuptools wheel
ARCHFLAGS="-arch arm64" uv pip install --no-build-isolation -e '.[geometry]'

Weights, placed under weights/ (paths are overridable by flags):

Component	Source	License
Pixal3D checkpoints	TencentARC/Pixal3D → weights/pixal3d	MIT
DINOv3 ViT-L/16	facebook/dinov3-vitl16-pretrain-lvd1689m (gated) → weights/conditioners/dinov3	DINOv3 License (Meta)
NAF upsampler	valeoai/NAF release, converted to safetensors → weights/conditioners/naf/model.safetensors	Apache-2.0
MoGe-2	Ruicheng/moge-2-vitl	See model card

The verified run used a byte-identical Pixal3D mirror (Pq234/Pixal3D, all seven checkpoint SHA-256s match Tencent's) and the FP16 DINOv3/NAF safetensors from ddalcu/Pixal3D-MLX-Serve-8bit. MoGe-2's model.pt was converted once to safetensors (format only).

Generate (close memory-heavy apps first on 16 GB):

PYTHONPATH=. python generate_pixal3d.py --image boot.png --output boot.glb \
  --resolution 512 --quantize 4 --texture-size 1024

Credits

Pixal3D: Tencent ARC Lab, Tsinghua University, Victoria University of Wellington. MLX port: lyonsno/pixal3d-mlx. Native Metal geometry backends: pawel-mazurkiewicz/Pixal3D-mac at 0be9e69. Not affiliated with or endorsed by the original authors.