TripoSG / README.md

Upload README.md

6673a9d verified 1 day ago

7.18 kB

license: mit
pipeline_tag: image-to-3d
base_model: VAST-AI/TripoSG
tags:
  - lux3d
  - rust
  - candle
  - safetensors
  - canonical-weights
  - 3d
  - image-to-3d
  - mesh
  - triposg

Lux3D TripoSG Canonical Weights

This repository contains the canonicalized safetensors export of the TripoSG image-to-3D checkpoint used by the Lux3D runtime.

Model Details

Model family: triposg
Canonical file: model.safetensors (7,946,503,588 bytes, about 7.4 GiB)
Tensor count: 1349
Parameter storage: F32
Primary runtime target: Lux3D runtime
Typical output: triangle mesh export (.obj)
SHA-256: 68c303df01728e2b7a48c185736b6d7e1fafd7ae98618552b72e51a00da55838

TripoSG is a high-fidelity image-to-3D foundation model built on large-scale rectified flow. This inference slice includes:

a DINOv2-large image encoder (24 layers, width 1024, patch 14) producing the conditioning embedding;
a 21-block rectified-flow DiT (width 2048, 16 heads, cross-attention width 1024, 2048 latent tokens);
a signed-distance VAE decoder (16 self-attention blocks plus one cross-attention block) that evaluates the SDF on a hierarchical grid;
hierarchical marching cubes at level 0 over the canonical ±1.005 bounds (dense depth 8, refinement depth 9 by default).

The source and target weights are MIT licensed.

Included Files

model.safetensors — merged canonical weights with image_encoder.*, transformer.*, and vae.* prefixes
resolved_config.json — runtime configuration extracted from the upstream configs
manifest.json — provenance, source checksums, tensor count, and raw-file list
checksums.json — SHA-256 and size for every file in this package

Usage With Lux3D

# Inspect the family contract
cargo run -p lux3d-cli -- inspect --repo-root <runtime-root> triposg

# Run with an explicit canonical package directory
cargo run -p lux3d-cli --features cuda -- run triposg --model-path <canonical-triposg-dir> --source <input-image> --output <output-file.obj>

The package can also be downloaded automatically into the Lux3D cache:

cargo run -p lux3d-cli --features cuda -- run triposg --source <input-image> --output <output-file.obj>

Default inference settings match the reference pipeline: 50 rectified-flow steps, CFG scale 7.0, seed 42, 2048 latent tokens, dense octree depth 8, hierarchical depth 9. Override them with --steps, --guidance-scale, --seed, --num-tokens, --dense-octree-depth, and --hierarchical-octree-depth.

A fast preview suitable for smoke tests:

cargo run -p lux3d-cli --features cuda -- run triposg --model-path <canonical-triposg-dir> --source <input-image> --steps 2 --dense-octree-depth 5 --hierarchical-octree-depth 5 --output <preview.obj>

Input should be a single image. RGBA images with a valid alpha channel use the reference foreground-bbox + white-composite + 10% padding path; RGB images (or invalid alpha) are treated as opaque over white. Background removal is intentionally not part of the Lux3D runtime.

Provenance

Upstream source model: VAST-AI/TripoSG
Upstream code: VAST-AI-Research/TripoSG (MIT)
Canonicalization flow: tools/python_baseline/normalize_weights.py

The raw checkpoint is split across transformer/, vae/, and image_encoder_dinov2/; normalization merges them into one safetensors file with family prefixes and records every source checksum in manifest.json. No tensor values are modified.

Validation

The Lux3D runtime was validated against the upstream PyTorch reference (F32, RTX 3060) on the golden sample:

Check	Result
preprocess uint8	exact
normalized `pixel_values` max delta	2.384e-7
DINOv2 `image_embeds` max delta	8.631e-5
step-0 CFG velocity max delta	8.631e-5
latents at step 1 / 10 / 25 max delta	1.67e-6 / 4.05e-6 / 9.84e-3
VAE SDF probe (baseline latents) max delta	1.78e-3
dense depth-8 SDF mean / p99.99	5.27e-6 / 2.94e-3
golden depth-9 mesh bbox / volume delta	1.72e-4 / 8.26e-5

The 50-step classifier-free-guided ODE is numerically chaotic: a control run of the same reference code on a different GPU/torch build drifts at least as much as Rust at the final step (reference-to-reference final latents: mean 1.42e-2, relative L2 7.74e-2, max 3.00). Final-state parity is therefore accepted statistically (mean ≤ 2e-2, relative L2 ≤ 1.2e-1, max ≤ 4.0), while early-step, VAE, grid, and mesh parity remain tight.

Reference timings on an NVIDIA GeForce RTX 3060 12 GiB (Windows/WDDM, release build):

Stage	CUDA (50 steps, depth 8→9)	Vulkan (2 steps, depth 5/5)	WGPU (2 steps, depth 5/5)
load	6.50 s	6.69 s	6.78 s
infer (preprocess + DINO + denoise)	118.25 s	11.97 s	24.25 s
SDF grid + mesh	178.54 s	0.64 s	0.79 s
export	0.48 s	0.00 s	0.00 s
peak VRAM	9.70 GiB	11.96 GiB	11.78 GiB

The default CUDA run produced 967,685 vertices and 1,932,352 faces. CPU, CUDA, Vulkan, and WGPU were exercised; Metal has no test host in this repository.

Intended Use

image-to-mesh generation with Lux3D
reproducible runtime validation against the upstream PyTorch reference
local, offline inference from canonical safetensors weights

Limitations

The upstream TripoSG code and weights are MIT licensed; review the upstream terms before redistribution.
The 50-step sampling trajectory is chaotic across frameworks and GPU environments. Final-state comparisons should use statistical tolerance, not bit-exact equality.
Lux3D's seeded latent initialization is deterministic but not bit-identical to the torch RNG; parity tests inject the baseline latents.
Background removal (RMBG-1.4 / BriaRMBG) is not implemented. Supply an image whose background is already white or transparent.
The flash decoder / DiffDMC path and the unused VAE encoder + quant weights are excluded; the contract records those exclusions.
Inference is F32-only in the current Lux3D runtime.
Metal inference is not tested in this package; CPU, CUDA, Vulkan, and WGPU are the verified backends.
All three submodels stay resident, so the native Vulkan/WGPU backends are very close to the 12 GiB limit; 16 GiB or more is recommended outside CUDA.
This package contains canonical runtime artifacts only, not the upstream training or demo stack.
The Lux3D contract JSON is shipped with the runtime, not inside this weight package.

License

Upstream model page: VAST-AI/TripoSG
Upstream code: VAST-AI-Research/TripoSG
Upstream license: MIT

Citation

@article{li2025triposg,
  title={TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models},
  author={Li, Yangguang and Zou, Zi-Xin and Liu, Zexiang and Wang, Dehu and Liang, Yuan and Yu, Zhipeng and Liu, Xingchao and Guo, Yuan-Chen and Liang, Ding and Ouyang, Wanli and others},
  journal={arXiv preprint arXiv:2502.06608},
  year={2025}
}