TRELLIS.2-4B / README.md
FerrisMind's picture
Upload 2 files
165dba0 verified
|
Raw History Blame Contribute Delete
10.7 kB
---
license: mit
pipeline_tag: image-to-3d
base_model: microsoft/TRELLIS.2-4B
tags:
- lux3d
- rust
- candle
- safetensors
- canonical-weights
- 3d
- image-to-3d
- mesh
- trellis2
---
# Lux3D TRELLIS.2-4B Canonical Weights
This repository contains the canonicalized `safetensors` export of the
TRELLIS.2-4B image-to-3D checkpoint used by the Lux3D runtime.
## Model Details
- Model family: `trellis2`
- Canonical file: `model.safetensors` (16,180,079,934 bytes, about 15.07 GiB)
- Tensor count: `4265`
- Parameter storage: mixed BF16 / F16 / F32, preserved from the upstream
checkpoints (`BF16`: 3200 tensors, `F16`: 612, `F32`: 453)
- SHA-256: `aaff394292cab30af64624e816eb3ac15901620b10b9e90f89754f47d05afee7`
- Raw provenance files: `19` (nine safetensors shards, their configs,
`pipeline.json`)
- Primary runtime target: Lux3D runtime
- Typical output: textured GLB mesh plus a PBR vertex-attribute sidecar
(`.obj` and `.ply` are also supported)
- Reference default pipeline: `1024_cascade` (512 sparse-structure + shape
SLAT pass, then the 1024 shape and texture passes)
TRELLIS.2-4B is Microsoft's 4-billion-parameter image-to-3D model built around
the O-Voxel sparse-voxel representation. This inference slice includes:
- the DINOv3 ViT-L/16 image conditioner (`image_cond`);
- a 64³ sparse-structure flow DiT and its dense 3D-convolution decoder
(`ss_flow`, `ss_dec`);
- 512 and 1024 shape SLAT flow DiTs (`shape_dit_512`, `shape_dit_1024`);
- the sparse shape VAE decoder that produces the O-Voxel geometry
(`shape_dec`);
- 512 and 1024 texture SLAT flow DiTs (`tex_dit_512`, `tex_dit_1024`);
- the sparse texture VAE decoder with per-voxel PBR attributes (`tex_dec`).
The merged package uses one `<stage>.` prefix per component:
| Stage prefix | Upstream source | Tensors |
|---|---|---:|
| `image_cond` | `external/dinov3/model.safetensors` | 415 |
| `ss_flow` | `ckpts/ss_flow_img_dit_1_3B_64_bf16.safetensors` | 640 |
| `ss_dec` | `external/ss_dec/ckpts/ss_dec_conv3d_16l8_fp16.safetensors` | 74 |
| `shape_dit_512` | `ckpts/slat_flow_img2shape_dit_1_3B_512_bf16.safetensors` | 640 |
| `shape_dit_1024` | `ckpts/slat_flow_img2shape_dit_1_3B_1024_bf16.safetensors` | 640 |
| `shape_dec` | `ckpts/shape_dec_next_dc_f16c32_fp16.safetensors` | 292 |
| `tex_dit_512` | `ckpts/slat_flow_imgshape2tex_dit_1_3B_512_bf16.safetensors` | 640 |
| `tex_dit_1024` | `ckpts/slat_flow_imgshape2tex_dit_1_3B_1024_bf16.safetensors` | 640 |
| `tex_dec` | `ckpts/tex_dec_next_dc_f16c32_fp16.safetensors` | 284 |
| **Total** | | **4265** |
## Included Files
- `model.safetensors` — merged canonical weights with the stage prefixes above
- `resolved_config.json` — `pipeline.json` arguments, every stage config, and
the shape/texture SLAT normalization vectors
- `manifest.json` — family, provenance, source checksums, raw-file order,
tensor count, and dtype histogram
- `checksums.json` — SHA-256 and byte size for the canonical data files
- `README.md` — this document
- `LICENSE` — MIT license text and third-party notices
No tensor values are modified by canonicalization: the nine shards are
merged tensor-by-tensor, keys gain their stage prefix, and original dtypes are
preserved byte-exact.
## Usage With Lux3D
Validate the family contract and run from an explicit canonical package:
```powershell
cargo run -p lux3d-cli -- inspect --repo-root <runtime-root> trellis2
cargo run -p lux3d-cli --features cuda -- run trellis2 `
--model-path <canonical-trellis2-dir> `
--rmbg-model-path <canonical-rmbg2-dir> `
--source <input-image> `
--output <output-file.glb>
```
A faster geometry-only preview:
```powershell
cargo run -p lux3d-cli --features cuda -- run trellis2 `
--model-path <canonical-trellis2-dir> `
--rmbg-model-path <canonical-rmbg2-dir> `
--source <input-image> `
--pipeline-type 512 --steps 12 --texture false --seed 42 `
--output <preview.obj>
```
`--pipeline-type` accepts `512` and `1024_cascade`. The single-resolution
`1024` path and the 1536-token cascade are not implemented by Lux3D yet.
The raw `microsoft/TRELLIS.2-4B` checkpoint directory (`pipeline.json` plus
`ckpts/` and `external/`) is accepted as-is by the CLI, so the raw tree does
not have to be normalized before a local run.
Auto-download is reserved for the `oxide-lab/TRELLIS.2-4B` repository. If that
repository is not available to your account, pass an explicit `--model-path`.
### Background removal
TRELLIS.2's reference preprocessing runs RMBG-2.0/BiRefNet when the input has
no usable alpha channel. Those weights are **not** part of this package.
Normalize them as the separate `rmbg2` family and pass
`--rmbg-model-path <canonical-rmbg2-dir>`, or place an `rmbg-2.0` directory
next to `--model-path`. A missing RMBG package is an error for RGB inputs;
Lux3D never silently treats a background as opaque. RGBA inputs with a usable
alpha channel use the alpha path and do not load RMBG.
The HTTP server accepts `--trellis2-rmbg-model-path` or
`LUX3D_TRELLIS2_RMBG_MODEL_PATH` and performs the same sibling discovery.
### Quantized variants
Standalone GGUF builds are published in
[`oxide-lab/TRELLIS.2-4B-GGUF`](https://huggingface.co/oxide-lab/TRELLIS.2-4B-GGUF):
| Variant | Storage | Size | Notes |
|---|---|---:|---|
| `f16` | F16, all tensors | 14.50 GiB | maximum fidelity |
| `q8_0` | Q8_0, F16 fallback | 8.87 GiB | recommended |
| `q6_k` | Q6_K, F16 fallback | 7.41 GiB | usable |
| `q5_k` | Q5_K, F16 fallback | 6.61 GiB | preview quality |
| `q4_k` | Q4_K, F16 fallback | 5.86 GiB | most aggressive |
Quantization is DiT-only: the five flow transformers use the `QMatMul` path;
DINOv3, the sparse VAE decoders, convolution kernels, norms, biases, and
embeddings stay in the F16 fallback. Each GGUF file embeds
`resolved_config.json`, so it is a complete `--model-path` target.
## Provenance
- Upstream model: [`microsoft/TRELLIS.2-4B`](https://huggingface.co/microsoft/TRELLIS.2-4B)
- Upstream code: [`microsoft/TRELLIS.2`](https://github.com/microsoft/TRELLIS.2) (MIT)
- Canonicalization flow: `tools/python_baseline/normalize_weights.py` in Lux3D
- Raw-file order and exact source checksums: `manifest.json`
Normalization reads the nine upstream safetensors shards plus their JSON
configs, `external/dinov3/config.json`, and `pipeline.json`. It merges them
into one file without materializing the full 15 GiB payload in memory. The
resulting package is verified by reopening one tensor per stage and comparing
shape, dtype, and bytes against the source shard.
## Validation
The Lux3D runtime was validated against the upstream PyTorch reference. Golden
stage checks:
| Check | Result |
|---|---:|
| DINOv3 conditioning relative error | 8.8e-7 |
| Sparse-VAE decoded vertices max delta | 5.96e-8 |
| Sparse-VAE decoded faces | exact |
| Sparse-structure decoder `coords_32` | exact |
| Shape SLAT final latent `rms_rel` | 1.17e-2 (upstream BF16 noise) |
Backend coverage on real weights (RTX 3060 12 GiB, 512 geometry smoke,
2 steps, synthetic RGBA input):
| Backend | Wall | Vertices / faces | Peak GPU (device) | Result |
|---|---:|---:|---:|---|
| CPU | 567 s | 3391 / 3044 | n/a | pass |
| CUDA | 23 s | 3391 / 3044 | 6.9 GiB | pass |
| Vulkan | 209 s | 3391 / 3044 | 7.8 GiB | pass |
| WGPU | 404 s | 3391 / 3044 | 9.6 GiB | pass |
Exported meshes from all four backends are finite and agree on topology and
bounding box to float noise; per-backend reruns reproduce the same counts and
byte-identical Vulkan/WGPU bounding boxes.
Full reference runs:
| Run | Result |
|---|---|
| CPU, 512, 12 steps | 424,289 vertices / 759,914 faces, 2221 s |
| CUDA, `1024_cascade`, 12 steps, texture on | 645,960 vertices / 636,302 faces, 438 s, ~7.8 GiB peak |
The CUDA cascade run was repeated and produced identical geometry counts and
byte-identical OBJ/PLY/GLB exports. Sequential stage loading keeps the full
textured cascade inside a 12 GiB card with roughly 4 GiB headroom.
Quantization fidelity was measured against this package on the same CUDA
backend and input (512 pipeline, 12 steps, seed 42, texture off). Drift is
relative to a reference bbox diagonal of 1.494:
| Variant | Chamfer-L1 rel | F@1% | Voxel IoU | Normal consistency |
|---|---:|---:|---:|---:|
| `f16` | 0.0027% | 1.000 | 0.996 | 0.993 |
| `q8_0` | 0.187% | 0.9835 | 0.623 | 0.904 |
| `q6_k` | 0.199% | 0.9813 | 0.609 | 0.905 |
| `q5_k` | 0.274% | 0.9729 | 0.445 | 0.890 |
| `q4_k` | 0.311% | 0.9625 | 0.400 | 0.885 |
The O-Voxel surface is discrete, so small latent shifts can flip individual
voxel occupancies; the Q8_0/Q6_K drift is mostly topology-level while
positions remain close (normals above 0.90). F16 reproduces the reference
mesh almost exactly.
## Intended Use
- image-to-mesh generation with Lux3D
- reproducible runtime validation against the upstream PyTorch reference
- local, offline inference from canonical safetensors weights
## Limitations
- Background removal requires the separate RMBG-2.0 package. It is not
included here and is licensed separately (CC-BY-NC-4.0, non-commercial).
- The single-resolution `1024` pipeline and the 1536-token cascade are not
implemented in Lux3D; use `512` or `1024_cascade`.
- Mesh post-processing (`fill_holes`, decimation) and texture-atlas UV
unwrapping are out of scope for the current runtime. Textured runs export
the O-Voxel dual-grid surface with per-vertex PBR colors in GLB plus a
lossless `<stem>.pbrvox.bin` + `<stem>.pbr.json` sidecar pair.
- One mesh is generated per image (`num_samples=1`).
- Metal inference is not tested on this Windows host. CPU, CUDA, Vulkan, and
WGPU are the verified backends.
- CPU inference works but is much slower than the GPU backends.
- Canonical weights are a runtime artifact only; this package does not contain
the upstream training or demo stack, and the Lux3D contract JSON ships with
the runtime rather than inside the weight package.
## License
TRELLIS.2 code and weights are released by Microsoft under the **MIT
License**; see [`LICENSE`](LICENSE). The DINOv3 ViT-L/16 encoder bundled
upstream under `image_cond.*` is a separately licensed gated Meta model;
verify its terms before redistribution or commercial use. RMBG-2.0 is a
separate BRIA model under CC-BY-NC-4.0 and is not part of this package.
## Citation
```bibtex
@article{xiang2025trellis2,
title={Native and Compact Structured Latents for 3D Generation},
author={Xiang, Jianfeng and Chen, Xiaoxue and Xu, Sicheng and Wang, Ruicheng and Lv, Zelong and Deng, Yu and Zhu, Hongyuan and Dong, Yue and Zhao, Hao and Yuan, Nicholas Jing and Yang, Jiaolong},
journal={Tech report},
year={2025}
}
```