Kaze-3D-v0.1

One image in, one 3D mesh out. A small image-to-3D model trained from scratch on a single RTX 3060 12 GB, written entirely in Rust on RustingBrain. No Python anywhere, in training or in use.

  • 4 seconds per mesh on an RTX 3060, about 2.1 GiB of card memory
  • runs on the CPU too
  • Apache-2.0 weights, trained only on CC-BY and CC0 data

Eight held-out objects: input image, true shape, Kaze output

Objects the model never saw: input image, true shape, Kaze's output. Chosen to show the range: the top four rows are among its best results of 64, the next two are above average, the last two are a typical and a weak one.

This is a first, early release. Kaze v0.1 gets the rough shape and size of an object and misses detail and exact proportions. It makes geometry only: no texture, no colour. See What to expect.

Quick start

You need a Rust toolchain. For the GPU build you also need an NVIDIA card and the CUDA toolkit.

# 1. the runner
git clone https://github.com/RustingStudio/RustingBrain
git clone https://github.com/RustingStudio/RustingRunner
cd RustingRunner
cargo build --release --features cuda      # leave out --features cuda for CPU

# 2. the weights, and the CLIP image encoder they need (its safetensors file
#    is in a pull request of the CLIP repository, hence --revision)
hf download rusting-studio/Kaze-3D-v0.1 --local-dir Kaze-3D-v0.1
hf download openai/clip-vit-base-patch16 model.safetensors --revision refs/pr/17 \
    --local-dir Kaze-3D-v0.1/clip-vit-base-patch16

# 3. one image to one mesh
./target/release/rusting-runner 3d --weights Kaze-3D-v0.1 \
    --image my-object.png --out my-object.glb --cuda 0

Leave out --cuda 0 to run on the CPU. Open the .glb in Blender or any glTF viewer. More flags (--seed, --steps, --guidance, --faces) are in the RustingRunner README.

Best input: one object, centred, on a plain or transparent background, seen slightly from above. 512ร—512 is ideal; other sizes are resized.

What to expect

  • The overall shape and size are usually right. Proportions are often a few percent off, and long objects sometimes come out turned about the vertical axis.
  • Shoes, boxes and flat objects work best. Thin parts, frames, round objects and objects made of several pieces are the weakest.
  • It was trained on renders of single objects. Photos, scenes and several objects in one image are outside what it has seen.
  • Geometry only. The mesh has no texture and no colour.

How it works

Part Size Role
Image encoder CLIP ViT-B/16, frozen, not in this repository turns the image into 197 tokens
Flow transformer 103,219,968 parameters turns noise into a shape latent, guided by the image tokens
Shape VAE 31,448,065 parameters turns the latent into a signed distance field; marching tetrahedra give the mesh

Measured quality

Measured on the RTX 3060 on Google Scanned Objects held out of training (held-out-ids.txt). The true shape is scaled so its longest side spans [-1, 1], and the output is scored as generated, with no alignment to the truth. F@0.02 is the F-score at a distance of 1% of the longest side, on 50,000 points per surface. Higher is better; 1.0 is a perfect match.

Objects F@0.02 IoU Our target (F@0.02)
Shape VAE, reconstruction 32 0.923 0.868 0.90, reached
Image to shape, 28 steps, guidance 4 64 0.215 0.344 0.55, not reached

6 of the 64 objects reach 0.55 on their own. One generation takes 4 seconds on the RTX 3060, weight loading included (three runs, 3.9 to 4.1 s).

Training data

112,364 objects, each rendered from 8 views:

  • Google Scanned Objects, 1,031 scans, CC-BY 4.0.
  • Objaverse, 111,333 objects: only those whose own licence is CC-BY 4.0 (110,699) or CC0 1.0 (634). Objects under non-commercial or share-alike licences were left out.

The shape VAE was trained on an earlier, smaller cut of the same corpus (53,688 objects, the same licences).

ATTRIBUTION.csv lists every Objaverse object used: id, title, author, licence and source page. Objaverse: Deitke et al., "Objaverse: A Universe of Annotated 3D Objects", CVPR 2023. Google Scanned Objects: Downs et al., ICRA 2022.

Training

  • Shape VAE: 225,000 steps, batch 16, learning rate 2e-4 decaying to 2e-5.
  • Flow transformer: 127,500 steps, batch 32, learning rate 1e-4 on a cosine towards 1e-5, about 490 ms a step. Training was stopped at 127,500 of a planned 200,000 steps. The released file is the averaged (EMA 0.9999) copy.
  • Hardware: one RTX 3060 12 GB.

Licence

Weights: Apache-2.0. The CLIP weights keep their own licence (MIT) and are not redistributed here.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support