--- license: mit pipeline_tag: image-to-3d base_model: microsoft/TRELLIS.2-4B tags: - lux3d - rust - candle - safetensors - canonical-weights - 3d - image-to-3d - mesh - trellis2 --- # Lux3D TRELLIS.2-4B Canonical Weights This repository contains the canonicalized `safetensors` export of the TRELLIS.2-4B image-to-3D checkpoint used by the Lux3D runtime. ## Model Details - Model family: `trellis2` - Canonical file: `model.safetensors` (16,180,079,934 bytes, about 15.07 GiB) - Tensor count: `4265` - Parameter storage: mixed BF16 / F16 / F32, preserved from the upstream checkpoints (`BF16`: 3200 tensors, `F16`: 612, `F32`: 453) - SHA-256: `aaff394292cab30af64624e816eb3ac15901620b10b9e90f89754f47d05afee7` - Raw provenance files: `19` (nine safetensors shards, their configs, `pipeline.json`) - Primary runtime target: Lux3D runtime - Typical output: textured GLB mesh plus a PBR vertex-attribute sidecar (`.obj` and `.ply` are also supported) - Reference default pipeline: `1024_cascade` (512 sparse-structure + shape SLAT pass, then the 1024 shape and texture passes) TRELLIS.2-4B is Microsoft's 4-billion-parameter image-to-3D model built around the O-Voxel sparse-voxel representation. This inference slice includes: - the DINOv3 ViT-L/16 image conditioner (`image_cond`); - a 64³ sparse-structure flow DiT and its dense 3D-convolution decoder (`ss_flow`, `ss_dec`); - 512 and 1024 shape SLAT flow DiTs (`shape_dit_512`, `shape_dit_1024`); - the sparse shape VAE decoder that produces the O-Voxel geometry (`shape_dec`); - 512 and 1024 texture SLAT flow DiTs (`tex_dit_512`, `tex_dit_1024`); - the sparse texture VAE decoder with per-voxel PBR attributes (`tex_dec`). The merged package uses one `.` prefix per component: | Stage prefix | Upstream source | Tensors | |---|---|---:| | `image_cond` | `external/dinov3/model.safetensors` | 415 | | `ss_flow` | `ckpts/ss_flow_img_dit_1_3B_64_bf16.safetensors` | 640 | | `ss_dec` | `external/ss_dec/ckpts/ss_dec_conv3d_16l8_fp16.safetensors` | 74 | | `shape_dit_512` | `ckpts/slat_flow_img2shape_dit_1_3B_512_bf16.safetensors` | 640 | | `shape_dit_1024` | `ckpts/slat_flow_img2shape_dit_1_3B_1024_bf16.safetensors` | 640 | | `shape_dec` | `ckpts/shape_dec_next_dc_f16c32_fp16.safetensors` | 292 | | `tex_dit_512` | `ckpts/slat_flow_imgshape2tex_dit_1_3B_512_bf16.safetensors` | 640 | | `tex_dit_1024` | `ckpts/slat_flow_imgshape2tex_dit_1_3B_1024_bf16.safetensors` | 640 | | `tex_dec` | `ckpts/tex_dec_next_dc_f16c32_fp16.safetensors` | 284 | | **Total** | | **4265** | ## Included Files - `model.safetensors` — merged canonical weights with the stage prefixes above - `resolved_config.json` — `pipeline.json` arguments, every stage config, and the shape/texture SLAT normalization vectors - `manifest.json` — family, provenance, source checksums, raw-file order, tensor count, and dtype histogram - `checksums.json` — SHA-256 and byte size for the canonical data files - `README.md` — this document - `LICENSE` — MIT license text and third-party notices No tensor values are modified by canonicalization: the nine shards are merged tensor-by-tensor, keys gain their stage prefix, and original dtypes are preserved byte-exact. ## Usage With Lux3D Validate the family contract and run from an explicit canonical package: ```powershell cargo run -p lux3d-cli -- inspect --repo-root trellis2 cargo run -p lux3d-cli --features cuda -- run trellis2 ` --model-path ` --rmbg-model-path ` --source ` --output ``` A faster geometry-only preview: ```powershell cargo run -p lux3d-cli --features cuda -- run trellis2 ` --model-path ` --rmbg-model-path ` --source ` --pipeline-type 512 --steps 12 --texture false --seed 42 ` --output ``` `--pipeline-type` accepts `512` and `1024_cascade`. The single-resolution `1024` path and the 1536-token cascade are not implemented by Lux3D yet. The raw `microsoft/TRELLIS.2-4B` checkpoint directory (`pipeline.json` plus `ckpts/` and `external/`) is accepted as-is by the CLI, so the raw tree does not have to be normalized before a local run. Auto-download is reserved for the `oxide-lab/TRELLIS.2-4B` repository. If that repository is not available to your account, pass an explicit `--model-path`. ### Background removal TRELLIS.2's reference preprocessing runs RMBG-2.0/BiRefNet when the input has no usable alpha channel. Those weights are **not** part of this package. Normalize them as the separate `rmbg2` family and pass `--rmbg-model-path `, or place an `rmbg-2.0` directory next to `--model-path`. A missing RMBG package is an error for RGB inputs; Lux3D never silently treats a background as opaque. RGBA inputs with a usable alpha channel use the alpha path and do not load RMBG. The HTTP server accepts `--trellis2-rmbg-model-path` or `LUX3D_TRELLIS2_RMBG_MODEL_PATH` and performs the same sibling discovery. ### Quantized variants Standalone GGUF builds are published in [`oxide-lab/TRELLIS.2-4B-GGUF`](https://huggingface.co/oxide-lab/TRELLIS.2-4B-GGUF): | Variant | Storage | Size | Notes | |---|---|---:|---| | `f16` | F16, all tensors | 14.50 GiB | maximum fidelity | | `q8_0` | Q8_0, F16 fallback | 8.87 GiB | recommended | | `q6_k` | Q6_K, F16 fallback | 7.41 GiB | usable | | `q5_k` | Q5_K, F16 fallback | 6.61 GiB | preview quality | | `q4_k` | Q4_K, F16 fallback | 5.86 GiB | most aggressive | Quantization is DiT-only: the five flow transformers use the `QMatMul` path; DINOv3, the sparse VAE decoders, convolution kernels, norms, biases, and embeddings stay in the F16 fallback. Each GGUF file embeds `resolved_config.json`, so it is a complete `--model-path` target. ## Provenance - Upstream model: [`microsoft/TRELLIS.2-4B`](https://huggingface.co/microsoft/TRELLIS.2-4B) - Upstream code: [`microsoft/TRELLIS.2`](https://github.com/microsoft/TRELLIS.2) (MIT) - Canonicalization flow: `tools/python_baseline/normalize_weights.py` in Lux3D - Raw-file order and exact source checksums: `manifest.json` Normalization reads the nine upstream safetensors shards plus their JSON configs, `external/dinov3/config.json`, and `pipeline.json`. It merges them into one file without materializing the full 15 GiB payload in memory. The resulting package is verified by reopening one tensor per stage and comparing shape, dtype, and bytes against the source shard. ## Validation The Lux3D runtime was validated against the upstream PyTorch reference. Golden stage checks: | Check | Result | |---|---:| | DINOv3 conditioning relative error | 8.8e-7 | | Sparse-VAE decoded vertices max delta | 5.96e-8 | | Sparse-VAE decoded faces | exact | | Sparse-structure decoder `coords_32` | exact | | Shape SLAT final latent `rms_rel` | 1.17e-2 (upstream BF16 noise) | Backend coverage on real weights (RTX 3060 12 GiB, 512 geometry smoke, 2 steps, synthetic RGBA input): | Backend | Wall | Vertices / faces | Peak GPU (device) | Result | |---|---:|---:|---:|---| | CPU | 567 s | 3391 / 3044 | n/a | pass | | CUDA | 23 s | 3391 / 3044 | 6.9 GiB | pass | | Vulkan | 209 s | 3391 / 3044 | 7.8 GiB | pass | | WGPU | 404 s | 3391 / 3044 | 9.6 GiB | pass | Exported meshes from all four backends are finite and agree on topology and bounding box to float noise; per-backend reruns reproduce the same counts and byte-identical Vulkan/WGPU bounding boxes. Full reference runs: | Run | Result | |---|---| | CPU, 512, 12 steps | 424,289 vertices / 759,914 faces, 2221 s | | CUDA, `1024_cascade`, 12 steps, texture on | 645,960 vertices / 636,302 faces, 438 s, ~7.8 GiB peak | The CUDA cascade run was repeated and produced identical geometry counts and byte-identical OBJ/PLY/GLB exports. Sequential stage loading keeps the full textured cascade inside a 12 GiB card with roughly 4 GiB headroom. Quantization fidelity was measured against this package on the same CUDA backend and input (512 pipeline, 12 steps, seed 42, texture off). Drift is relative to a reference bbox diagonal of 1.494: | Variant | Chamfer-L1 rel | F@1% | Voxel IoU | Normal consistency | |---|---:|---:|---:|---:| | `f16` | 0.0027% | 1.000 | 0.996 | 0.993 | | `q8_0` | 0.187% | 0.9835 | 0.623 | 0.904 | | `q6_k` | 0.199% | 0.9813 | 0.609 | 0.905 | | `q5_k` | 0.274% | 0.9729 | 0.445 | 0.890 | | `q4_k` | 0.311% | 0.9625 | 0.400 | 0.885 | The O-Voxel surface is discrete, so small latent shifts can flip individual voxel occupancies; the Q8_0/Q6_K drift is mostly topology-level while positions remain close (normals above 0.90). F16 reproduces the reference mesh almost exactly. ## Intended Use - image-to-mesh generation with Lux3D - reproducible runtime validation against the upstream PyTorch reference - local, offline inference from canonical safetensors weights ## Limitations - Background removal requires the separate RMBG-2.0 package. It is not included here and is licensed separately (CC-BY-NC-4.0, non-commercial). - The single-resolution `1024` pipeline and the 1536-token cascade are not implemented in Lux3D; use `512` or `1024_cascade`. - Mesh post-processing (`fill_holes`, decimation) and texture-atlas UV unwrapping are out of scope for the current runtime. Textured runs export the O-Voxel dual-grid surface with per-vertex PBR colors in GLB plus a lossless `.pbrvox.bin` + `.pbr.json` sidecar pair. - One mesh is generated per image (`num_samples=1`). - Metal inference is not tested on this Windows host. CPU, CUDA, Vulkan, and WGPU are the verified backends. - CPU inference works but is much slower than the GPU backends. - Canonical weights are a runtime artifact only; this package does not contain the upstream training or demo stack, and the Lux3D contract JSON ships with the runtime rather than inside the weight package. ## License TRELLIS.2 code and weights are released by Microsoft under the **MIT License**; see [`LICENSE`](LICENSE). The DINOv3 ViT-L/16 encoder bundled upstream under `image_cond.*` is a separately licensed gated Meta model; verify its terms before redistribution or commercial use. RMBG-2.0 is a separate BRIA model under CC-BY-NC-4.0 and is not part of this package. ## Citation ```bibtex @article{xiang2025trellis2, title={Native and Compact Structured Latents for 3D Generation}, author={Xiang, Jianfeng and Chen, Xiaoxue and Xu, Sicheng and Wang, Ruicheng and Lv, Zelong and Deng, Yu and Zhu, Hongyuan and Dong, Yue and Zhao, Hao and Yuan, Nicholas Jing and Yang, Jiaolong}, journal={Tech report}, year={2025} } ```