TripoSG / README.md
FerrisMind's picture
Upload README.md
6673a9d verified
|
Raw History Blame Contribute Delete
7.18 kB
---
license: mit
pipeline_tag: image-to-3d
base_model: VAST-AI/TripoSG
tags:
- lux3d
- rust
- candle
- safetensors
- canonical-weights
- 3d
- image-to-3d
- mesh
- triposg
---
# Lux3D TripoSG Canonical Weights
This repository contains the canonicalized `safetensors` export of the TripoSG image-to-3D checkpoint used by the Lux3D runtime.
## Model Details
- Model family: `triposg`
- Canonical file: `model.safetensors` (7,946,503,588 bytes, about 7.4 GiB)
- Tensor count: `1349`
- Parameter storage: F32
- Primary runtime target: Lux3D runtime
- Typical output: triangle mesh export (`.obj`)
- SHA-256: `68c303df01728e2b7a48c185736b6d7e1fafd7ae98618552b72e51a00da55838`
TripoSG is a high-fidelity image-to-3D foundation model built on large-scale rectified flow. This inference slice includes:
- a DINOv2-large image encoder (24 layers, width 1024, patch 14) producing the conditioning embedding;
- a 21-block rectified-flow DiT (width 2048, 16 heads, cross-attention width 1024, 2048 latent tokens);
- a signed-distance VAE decoder (16 self-attention blocks plus one cross-attention block) that evaluates the SDF on a hierarchical grid;
- hierarchical marching cubes at level 0 over the canonical `±1.005` bounds (dense depth 8, refinement depth 9 by default).
The source and target weights are MIT licensed.
## Included Files
- `model.safetensors` — merged canonical weights with `image_encoder.*`, `transformer.*`, and `vae.*` prefixes
- `resolved_config.json` — runtime configuration extracted from the upstream configs
- `manifest.json` — provenance, source checksums, tensor count, and raw-file list
- `checksums.json` — SHA-256 and size for every file in this package
## Usage With Lux3D
```powershell
# Inspect the family contract
cargo run -p lux3d-cli -- inspect --repo-root <runtime-root> triposg
# Run with an explicit canonical package directory
cargo run -p lux3d-cli --features cuda -- run triposg --model-path <canonical-triposg-dir> --source <input-image> --output <output-file.obj>
```
The package can also be downloaded automatically into the Lux3D cache:
```powershell
cargo run -p lux3d-cli --features cuda -- run triposg --source <input-image> --output <output-file.obj>
```
Default inference settings match the reference pipeline: 50 rectified-flow steps, CFG scale 7.0, seed 42, 2048 latent tokens, dense octree depth 8, hierarchical depth 9. Override them with `--steps`, `--guidance-scale`, `--seed`, `--num-tokens`, `--dense-octree-depth`, and `--hierarchical-octree-depth`.
A fast preview suitable for smoke tests:
```powershell
cargo run -p lux3d-cli --features cuda -- run triposg --model-path <canonical-triposg-dir> --source <input-image> --steps 2 --dense-octree-depth 5 --hierarchical-octree-depth 5 --output <preview.obj>
```
Input should be a single image. RGBA images with a valid alpha channel use the reference foreground-bbox + white-composite + 10% padding path; RGB images (or invalid alpha) are treated as opaque over white. Background removal is intentionally not part of the Lux3D runtime.
## Provenance
- Upstream source model: [`VAST-AI/TripoSG`](https://huggingface.co/VAST-AI/TripoSG)
- Upstream code: [`VAST-AI-Research/TripoSG`](https://github.com/VAST-AI-Research/TripoSG) (MIT)
- Canonicalization flow: `tools/python_baseline/normalize_weights.py`
The raw checkpoint is split across `transformer/`, `vae/`, and `image_encoder_dinov2/`; normalization merges them into one safetensors file with family prefixes and records every source checksum in `manifest.json`. No tensor values are modified.
## Validation
The Lux3D runtime was validated against the upstream PyTorch reference (F32, RTX 3060) on the golden sample:
| Check | Result |
|---|---:|
| preprocess uint8 | exact |
| normalized `pixel_values` max delta | 2.384e-7 |
| DINOv2 `image_embeds` max delta | 8.631e-5 |
| step-0 CFG velocity max delta | 8.631e-5 |
| latents at step 1 / 10 / 25 max delta | 1.67e-6 / 4.05e-6 / 9.84e-3 |
| VAE SDF probe (baseline latents) max delta | 1.78e-3 |
| dense depth-8 SDF mean / p99.99 | 5.27e-6 / 2.94e-3 |
| golden depth-9 mesh bbox / volume delta | 1.72e-4 / 8.26e-5 |
The 50-step classifier-free-guided ODE is numerically chaotic: a control run of the same reference code on a different GPU/torch build drifts at least as much as Rust at the final step (reference-to-reference final latents: mean 1.42e-2, relative L2 7.74e-2, max 3.00). Final-state parity is therefore accepted statistically (mean ≤ 2e-2, relative L2 ≤ 1.2e-1, max ≤ 4.0), while early-step, VAE, grid, and mesh parity remain tight.
Reference timings on an NVIDIA GeForce RTX 3060 12 GiB (Windows/WDDM, release build):
| Stage | CUDA (50 steps, depth 8→9) | Vulkan (2 steps, depth 5/5) | WGPU (2 steps, depth 5/5) |
|---|---:|---:|---:|
| load | 6.50 s | 6.69 s | 6.78 s |
| infer (preprocess + DINO + denoise) | 118.25 s | 11.97 s | 24.25 s |
| SDF grid + mesh | 178.54 s | 0.64 s | 0.79 s |
| export | 0.48 s | 0.00 s | 0.00 s |
| peak VRAM | 9.70 GiB | 11.96 GiB | 11.78 GiB |
The default CUDA run produced 967,685 vertices and 1,932,352 faces. CPU, CUDA, Vulkan, and WGPU were exercised; Metal has no test host in this repository.
## Intended Use
- image-to-mesh generation with Lux3D
- reproducible runtime validation against the upstream PyTorch reference
- local, offline inference from canonical safetensors weights
## Limitations
- The upstream TripoSG code and weights are MIT licensed; review the upstream terms before redistribution.
- The 50-step sampling trajectory is chaotic across frameworks and GPU environments. Final-state comparisons should use statistical tolerance, not bit-exact equality.
- Lux3D's seeded latent initialization is deterministic but not bit-identical to the torch RNG; parity tests inject the baseline latents.
- Background removal (RMBG-1.4 / BriaRMBG) is not implemented. Supply an image whose background is already white or transparent.
- The flash decoder / DiffDMC path and the unused VAE encoder + quant weights are excluded; the contract records those exclusions.
- Inference is F32-only in the current Lux3D runtime.
- Metal inference is not tested in this package; CPU, CUDA, Vulkan, and WGPU are the verified backends.
- All three submodels stay resident, so the native Vulkan/WGPU backends are very close to the 12 GiB limit; 16 GiB or more is recommended outside CUDA.
- This package contains canonical runtime artifacts only, not the upstream training or demo stack.
- The Lux3D contract JSON is shipped with the runtime, not inside this weight package.
## License
- Upstream model page: [`VAST-AI/TripoSG`](https://huggingface.co/VAST-AI/TripoSG)
- Upstream code: [`VAST-AI-Research/TripoSG`](https://github.com/VAST-AI-Research/TripoSG)
- Upstream license: MIT
## Citation
```bibtex
@article{li2025triposg,
title={TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models},
author={Li, Yangguang and Zou, Zi-Xin and Liu, Zexiang and Wang, Dehu and Liang, Yuan and Yu, Zhipeng and Liu, Xingchao and Guo, Yuan-Chen and Liang, Ding and Ouyang, Wanli and others},
journal={arXiv preprint arXiv:2502.06608},
year={2025}
}
```