ucalyptus/trellis-image-large-mlx

TRELLIS image-large — MLX, verified on Apple M3

Converted official TRELLIS image-large checkpoints for the experimental native MLX image-to-Gaussian pipeline in mlx-diffuser. No fine-tuning or retraining was performed.

Output is a 3D Gaussian-splat PLY, not a triangle mesh or textured GLB. This checkpoint does not include the mesh/FlexiCubes decoder, radiance-field decoder, or training encoders.

Included components

Directory	Source	Conversion
image_conditioner	facebook/dinov2-with-registers-large	8-bit, group size 64
sparse_structure_flow	ss_flow_img_dit_L_16l8_fp16	8-bit, group size 64
sparse_structure_decoder	ss_dec_conv3d_16l8_fp16	Original mixed precision; MLX Conv3D layout
slat_flow	slat_flow_img_dit_L_64l8p2_fp16	Original mixed precision; native sparse MLX/Metal operations
gaussian_decoder	slat_dec_gs_swin8_B_64l8gs32_fp16	Original mixed precision; native sparse Swin decoder

The trellis.json manifest is required by the MLX runtime. These are converted MLX weights, not a directory intended for the original CUDA pipeline.

Measured local verification

The sample in examples/ was actually generated and checked on an Apple M3 with a 10-core integrated GPU and 16 GB unified memory, using MLX 0.31.2 and Python 3.11.11. The runtime used the GPU/Metal backend with staged low-memory evaluation.

Measurement	Observed result
Download plus checkpoint conversion	42.18 seconds
Generation, excluding PLY export	272.78 seconds
PLY export	1.29 seconds
Sampling	25 structure + 25 SLAT steps, seed 42
Occupied voxels	13,542
Gaussian splats	433,344
MLX peak allocation during generation	2,151,160,152 bytes (2.00 GiB)
Peak process footprint, macOS /usr/bin/time -l	3,182,954,200 bytes (2.96 GiB)
Peak resident set size	1,019,494,400 bytes (0.95 GiB)
Original ASCII PLY size	70,667,908 bytes

MLX allocation, resident set size, and process footprint are different measurements; none is total system memory consumption. These measurements cover one sample and are not universal memory or runtime limits. Initial conversion is excluded from generation time.

Verification checked the PLY header and all 17 parameter fields, finite values, positive covariance scales, unit rotation quaternions, and agreement between decoded voxel and exported Gaussian counts. All 433,344 splats were loaded and rendered in Chrome with GaussianSplats3D; four fully depth-sorted viewpoints were inspected. examples/metrics.json, examples/conversion-metrics.json, and examples/validation.json contain the measurements.

Run locally

Use Apple Silicon and Python 3.11+. The exact converter/runtime source revision used here was a26b42aee4e31999dbb4429226b66d896d49e1d8 (mlx-diffuser 0.1.6).

pip install "mlx==0.31.2" "mlx-diffuser[trellis] @ git+https://github.com/amirhossein-razlighi/mlx_diffuser.git@a26b42aee4e31999dbb4429226b66d896d49e1d8"

from huggingface_hub import snapshot_download
from mlx_diffuser import TrellisImageTo3DPipeline

# If the repository is private, authenticate using your own HF_TOKEN or hf auth login.
checkpoint = snapshot_download(
    "ucalyptus/trellis-image-large-mlx",
    ignore_patterns=["examples/*"],
)
pipeline = TrellisImageTo3DPipeline.from_pretrained(checkpoint)
result = pipeline(
    "object.png",  # Prefer an isolated object in a transparent PNG.
    seed=42,
    sparse_structure_steps=25,
    slat_steps=25,
    low_memory=True,
    remove_background=False,
)
result.save_ply("object.ply")

The pipeline loads and releases components stage by stage. Images with opaque backgrounds should be segmented before inference or processed using the runtime's optional background-removal dependencies.

Limitations

• Experimental MLX implementation; component-by-component numerical parity against the PyTorch/CUDA reference remains pending.
• One input image; hidden geometry is generated rather than measured.
• Gaussian splats are not a watertight or fabrication-ready mesh.
• The native MLX package does not include the CUDA reference renderer. The previews here use a separate WebGL Gaussian renderer and do not establish pixel parity with CUDA rendering.
• These are original TRELLIS-image-large components, not the TRELLIS.2-4B pipeline.

Provenance and licenses

• TRELLIS weights: Microsoft TRELLIS-image-large, MIT License, copyright Microsoft Corporation.
• DINOv2 conditioner: facebook/dinov2-with-registers-large, Apache License 2.0.
• Conversion/runtime: mlx-diffuser, original code CC0-1.0; its code is not bundled in this checkpoint.
• Input boot image: the runtime project's docs/assets/trellis_boot_input.png, identified there as a synthetic photorealistic example.
• Preview rendering: GaussianSplats3D, MIT; viewer code is not bundled in this model repository.

The upstream MIT and Apache 2.0 license texts are preserved in LICENSE. Conversion and quantization do not replace the upstream model licenses. See NOTICE for the changes made to the original model files.