Update model card: description, intended use, limitations

README.md

@@ -3,9 +3,14 @@ license: mit
 license_link: LICENSE
 extra_gated_eu_disallowed: true
 pipeline_tag: image-to-3d
+tags:
+  - 3d-generation
+  - image-to-3d
+  - pixel-aligned
+  - pbr-textures
+  - game-assets
 ---
 
-
 <div align="center">
 
 # Pixal3D: Pixel-Aligned 3D Generation from Images
@@ -29,250 +34,115 @@ pipeline_tag: image-to-3d
   <a href="LICENSE"><img src=https://img.shields.io/badge/License-MIT-yellow.svg height=22px></a>
 </div>
 
-
-**Pixal3D** generates high-fidelity 3D assets from a single image. Unlike previous methods that loosely inject image features via attention, Pixal3D explicitly lifts pixel features into 3D through back-projection, establishing direct pixel-to-3D correspondences. This enables near-reconstruction-level fidelity with detailed geometry and PBR textures.
-
----
-
-## ✨ News
-
-- **May 2026**: Release training code and data preparation toolkit. 🔧
-- **May 2026**: Release the improved version based on [Trellis.2](https://github.com/microsoft/TRELLIS.2) backbone. 💪
-- **May 2026**: Release inference code and online demo. 🤗
-- **Apr 2026**: Our paper is accepted to SIGGRAPH 2026! 🎉
-
-## 📌 Branches
-
-| Branch | Description |
-|--------|-------------|
-| `main` | **Latest version** — improved implementation based on [Trellis.2](https://github.com/microsoft/TRELLIS.2) backbone with better performance. |
-| `paper` | **Paper version** — original implementation based on [Direct3D-S2](https://github.com/DreamTechAI/Direct3D-S2), corresponding to results reported in our SIGGRAPH 2026 paper. |
-
-> If you want to reproduce the results in our paper, please switch to the `paper` branch.
-
-## 🎮 Try It Online
-
-You can try Pixal3D directly in your browser without any installation via our Hugging Face Gradio demo:
-
-👉 [**Launch Demo**](https://huggingface.co/spaces/TencentARC/Pixal3D)
-
-## 🚀 Getting Started
-
-### Installation
-
-#### Step 1: Follow TRELLIS.2 Installation
-
-Please first follow the installation guide of [TRELLIS.2](https://github.com/microsoft/TRELLIS.2) to set up the base environment.
-
-#### Step 2: Install Additional Dependencies
-
-```bash
-pip install -r requirements.txt
-```
-
-#### Step 3: Install natten
-
-```bash
-NATTEN_CUDA_ARCH="xx" NATTEN_N_WORKERS=xx pip install natten==0.21.0 --no-build-isolation
-```
-
-Please replace `xx` with the CUDA architecture and the number of build workers suitable for your machine.
-
-#### Step 4: Install utils3d
-
-```bash
-pip install https://github.com/LDYang694/Storages/releases/download/20260430/utils3d-0.0.2-py3-none-any.whl
-```
-
-> **Note**: `requirements-hfdemo.txt` is for the Hugging Face Spaces demo (H-series GPU architecture) and may not be compatible with other architectures.
-
-### Usage
-
-#### Inference
-
-Generate a GLB mesh from a single image:
+## TL;DR
+
+**One image in, a textured 3D asset out.** Pixal3D generates high-fidelity 3D assets
+(geometry + PBR textures, exported as GLB) from a single image, and extends naturally to
+multi-view turnarounds and object-separated scenes. Its defining trait is **pixel
+alignment**: the 3D generation is conditioned on pixels lifted directly into 3D space, so
+the surface where your image's chest logo sits carries that logo — fidelity approaches
+reconstruction rather than plausible imitation.
+
+## Model description
+
+Most 3D-native generators synthesize shape in a canonical space and inject image cues
+loosely via attention, leaving the pixel-to-3D association ambiguous. Pixal3D instead
+generates **in a pixel-aligned way, consistent with the input view**: a pixel
+back-projection conditioning scheme lifts multi-scale image features into a 3D feature
+volume, establishing direct pixel-to-3D correspondence ([arXiv:2605.10922](https://arxiv.org/abs/2605.10922)).
+
+- **Backbone**: improved implementation on the [TRELLIS.2](https://github.com/microsoft/TRELLIS.2)
+  backbone (this `main` branch); the original SIGGRAPH paper implementation is on the
+  `paper` branch ([Direct3D-S2](https://github.com/DreamTechAI/Direct3D-S2)-based).
+- **Architecture**: a three-stage cascade of flow-matching DiTs (1.3B each) —
+  sparse structure → shape (SLAT) → texture (PBR) — each trained at progressively higher
+  resolutions and conditioned by pixel back-projection with view-aligned latents.
+- **Image encoder**: DINOv2 (Apache-2.0, see [NOTICE](NOTICE)).
+- **Output**: GLB with detailed geometry and PBR (metallic-roughness) textures.
+- **Multi-view**: aggregating back-projected feature volumes across views improves
+  fidelity to the surfaces every view shares.
+- **Scenes**: the same pixel-aligned generation composes into object-separated 3D scenes
+  via a modular pipeline.
+
+## Intended use
+
+- Turning a single concept image, photo or render into a usable 3D asset for games,
+  rendering, AR/VR, design iteration and 3D content prototyping.
+- Multi-view inputs (e.g. turnaround sheets) when surfaces must match several views.
+- Scene synthesis from an image containing multiple objects.
+
+## Limitations and biases
+
+- **Generated, not measured.** Output is a generated asset, not a 3D scan; do not use it
+  where metric accuracy matters.
+- **Unseen is invented.** Surfaces no view can see are hallucinated by the model;
+  multi-view inputs reduce this but do not eliminate it.
+- **Training-data bias.** Trained on ObjaverseXL (Sketchfab subset), which is
+  object-centric and dominated by render-style imagery: everyday photographed objects
+  and heavily stylised flat/vector art can come out weaker than render-style inputs.
+- **Generation pose.** Pixal3D generates in the input view's alignment rather than a
+  canonical pose; pipelines expecting canonical-front assets should account for this.
+- **Hardware.** Standard mode (pipeline resolution 1536) targets high-VRAM NVIDIA GPUs;
+  `--low_vram` (resolution 1024, on-demand loading) trades peak VRAM down. The online
+  demo runs on H-series GPUs.
+
+## Quick start
+
+Full instructions live on [GitHub](https://github.com/TencentARC/Pixal3D). In short:
+set up the TRELLIS.2 environment, then `pip install -r requirements.txt`, build
+`natten==0.21.0` for your CUDA architecture, and install the provided `utils3d` wheel.
+
+Generate a GLB from one image:
 
 ```bash
 python inference.py --image assets/images/0_img.png --output ./output.glb
 ```
 
-**Low-VRAM mode** (reduces peak VRAM by loading models on-demand):
+Lower peak VRAM (loads models on-demand, default resolution 1024 instead of 1536):
 
 ```bash
 python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram
 ```
 
-By default, the pipeline resolution is **1536** (standard mode) or **1024** (low-VRAM mode). You can override this with `--resolution`:
+Without `flash_attn`, PyTorch's SDPA backend works too:
 
 ```bash
-# Force 1536 even in low-VRAM mode
-python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram --resolution 1536
-
-# Force 1024 in standard mode
-python inference.py --image assets/images/0_img.png --output ./output.glb --resolution 1024
+ATTN_BACKEND=sdpa python inference.py --image assets/images/0_img.png --output ./output.glb
 ```
 
-**Tip**: If you don't have `flash_attn` installed, you can use PyTorch's built-in SDPA backend instead:
-> ```bash
-> ATTN_BACKEND=sdpa python inference.py --image assets/images/0_img.png --output ./output.glb --low_vram
-> ```
+A Gradio web demo is included (`python app.py`, also with `--low_vram`), and a hosted
+demo runs at the [Hugging Face Space](https://huggingface.co/spaces/TencentARC/Pixal3D).
 
-### Web Demo
-
-We provide a Gradio web demo for Pixal3D, which allows you to generate 3D meshes from images interactively.
+## 📌 Branches
 
-```bash
-python app.py 
-```
+| Branch | Description |
+|--------|-------------|
+| `main` | **Latest version** — improved implementation based on [TRELLIS.2](https://github.com/microsoft/TRELLIS.2) backbone with better performance. |
+| `paper` | **Paper version** — original implementation based on [Direct3D-S2](https://github.com/DreamTechAI/Direct3D-S2), corresponding to results reported in our SIGGRAPH 2026 paper. |
 
-Low-VRAM mode is also available for the web demo. The frontend default resolution will automatically switch to 1024 in low-VRAM mode (1536 otherwise), but can be changed manually in the UI.
+> If you want to reproduce the results in our paper, please switch to the `paper` branch.
 
-```bash
-python app.py --low_vram
-# or via environment variable:
-LOW_VRAM=1 python app.py
-```
 ## 🔧 Training
 
-We provide the full training codebase for reproducing Pixal3D from scratch.
-
-### Data Preparation
-
-Prepare view-aligned O-Voxel data and rendered condition images by following the data toolkit instructions:
+The full training codebase (three-stage cascade, ObjaverseXL data toolkit, per-stage
+configs and the complete argument reference) is on
+[GitHub](https://github.com/TencentARC/Pixal3D#-training).
 
-> 📂 **[data_toolkit/README.md](data_toolkit/README.md)**
+| Stage | Model | Resolutions |
+|-------|-------|-------------|
+| 1 | Sparse Structure | 32 → 64 |
+| 2 | Shape | 256 → 512 → 1024 |
+| 3 | Texture (PBR) | 256 → 512 → 1024 |
 
-### Overview
-
-Pixal3D is trained as a three-stage cascade, each progressively increasing resolution:
-
-| Stage | Model | Resolutions | Config Prefix |
-|-------|-------|-------------|---------------|
-| 1 | Sparse Structure | 32 → 64 | `ss_flow_img_dit_*_proj_finetune` |
-| 2 | Shape | 256 → 512 → 1024 | `slat_flow_img2shape_*_proj_finetune` |
-| 3 | Texture | 256 → 512 → 1024 | `slat_flow_imgshape2tex_*_proj_finetune` |
-
-All stages use **pixel-aligned projection conditioning** and **view-aligned latents** (2 views by default). Within each stage, start from the lowest resolution and progressively fine-tune to higher resolutions by setting `finetune_ckpt` in the config.
-
-### Quick Start
-
-```sh
-python train.py \
-  --config <CONFIG_JSON> \
-  --output_dir <OUTPUT_DIR> \
-  --data_dir '<DATA_DIR_JSON>'
-```
-
-`--data_dir` is a JSON string describing the dataset layout. Different stages require different keys:
-
-| Stage | Required keys |
-|-------|---------------|
-| Sparse Structure | `base`, `ss_latent`, `render_cond` |
-| Shape | `base`, `shape_latent`, `render_cond` |
-| Texture | `base`, `shape_latent`, `pbr_latent`, `render_cond` |
-
-### Example: Training All Three Stages
-
-Below we show the full training sequence using ObjaverseXL as an example. Each higher-resolution step requires updating `finetune_ckpt` in its config JSON to point to the previous checkpoint.
-
-<details>
-<summary><b>Stage 1: Sparse Structure (32 → 64)</b></summary>
-
-```sh
-# Resolution 32
-python train.py \
-  --config configs/gen/ss_flow_img_dit_1_3B_32_bf16_proj_finetune.json \
-  --output_dir results/ss_32 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "ss_latent": "datasets/ObjaverseXL_sketchfab/ss_latents/ss_enc_conv3d_16l8_fp16_64_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-
-# Resolution 64 (set finetune_ckpt → results/ss_32 checkpoint)
-python train.py \
-  --config configs/gen/ss_flow_img_dit_1_3B_32_bf16_proj_finetune_ft64.json \
-  --output_dir results/ss_ft64 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "ss_latent": "datasets/ObjaverseXL_sketchfab/ss_latents/ss_enc_conv3d_16l8_fp16_64_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-```
-</details>
-
-<details>
-<summary><b>Stage 2: Shape (256 → 512 → 1024)</b></summary>
-
-```sh
-# Resolution 256
-python train.py \
-  --config configs/gen/slat_flow_img2shape_dit_1_3B_256_bf16_proj_finetune.json \
-  --output_dir results/shape_256 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "shape_latent": "datasets/ObjaverseXL_sketchfab/shape_latents/shape_enc_next_dc_f16c32_fp16_256_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-
-# Resolution 512
-python train.py \
-  --config configs/gen/slat_flow_img2shape_dit_1_3B_256_bf16_proj_finetune_ft512.json \
-  --output_dir results/shape_ft512 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "shape_latent": "datasets/ObjaverseXL_sketchfab/shape_latents/shape_enc_next_dc_f16c32_fp16_512_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-
-# Resolution 1024
-python train.py \
-  --config configs/gen/slat_flow_img2shape_dit_1_3B_512_bf16_proj_finetune_ft1024.json \
-  --output_dir results/shape_ft1024 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "shape_latent": "datasets/ObjaverseXL_sketchfab/shape_latents/shape_enc_next_dc_f16c32_fp16_1024_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-```
-</details>
-
-<details>
-<summary><b>Stage 3: Texture (256 → 512 → 1024)</b></summary>
-
-```sh
-# Resolution 256
-python train.py \
-  --config configs/gen/slat_flow_imgshape2tex_dit_1_3B_256_bf16_proj_finetune.json \
-  --output_dir results/tex_256 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "shape_latent": "datasets/ObjaverseXL_sketchfab/shape_latents/shape_enc_next_dc_f16c32_fp16_256_view", "pbr_latent": "datasets/ObjaverseXL_sketchfab/pbr_latents/tex_enc_next_dc_f16c32_fp16_256_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-
-# Resolution 512
-python train.py \
-  --config configs/gen/slat_flow_imgshape2tex_dit_1_3B_512_bf16_proj_finetune.json \
-  --output_dir results/tex_512 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "shape_latent": "datasets/ObjaverseXL_sketchfab/shape_latents/shape_enc_next_dc_f16c32_fp16_512_view", "pbr_latent": "datasets/ObjaverseXL_sketchfab/pbr_latents/tex_enc_next_dc_f16c32_fp16_512_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-
-# Resolution 1024
-python train.py \
-  --config configs/gen/slat_flow_imgshape2tex_dit_1_3B_512_bf16_proj_finetune_ft1024.json \
-  --output_dir results/tex_ft1024 \
-  --data_dir '{"ObjaverseXL_sketchfab": {"base": "datasets/ObjaverseXL_sketchfab", "shape_latent": "datasets/ObjaverseXL_sketchfab/shape_latents/shape_enc_next_dc_f16c32_fp16_1024_view", "pbr_latent": "datasets/ObjaverseXL_sketchfab/pbr_latents/tex_enc_next_dc_f16c32_fp16_1024_view", "render_cond": "datasets/ObjaverseXL_sketchfab/renders_cond"}}'
-```
-</details>
-
-### Additional Options
-
-<details>
-<summary><b>All command-line arguments</b></summary>
-
-| Argument | Description | Default |
-|----------|-------------|---------|
-| `--config` | Config JSON path | *required* |
-| `--output_dir` | Output directory | *required* |
-| `--data_dir` | Dataset JSON string | `./data/` |
-| `--load_dir` | Checkpoint load directory | `output_dir` |
-| `--ckpt` | Resume from step | `latest` |
-| `--auto_retry` | Retries on failure | `3` |
-| `--tryrun` | Dry run | `false` |
-| `--profile` | Profiling | `false` |
-| `--num_nodes` | Number of nodes | `1` |
-| `--node_rank` | Current node rank | `0` |
-| `--num_gpus` | GPUs per node | all |
-| `--master_addr` | Master address | `localhost` |
-| `--master_port` | Master port | `12666` |
-| `--use_wandb` | Enable W&B logging | `false` |
-| `--wandb_project` | W&B project | `trellis2-training` |
-| `--wandb_name` | W&B run name | basename of `output_dir` |
-| `--wandb_id` | W&B run ID (resume) | — |
-
-</details>
+All stages use pixel-aligned projection conditioning and view-aligned latents (2 views
+by default), fine-tuned progressively from the lowest resolution within each stage.
 
 ## 🌐 Community Projects
 
 We thank the community for building extensions and deployment guides for Pixal3D!
 
 - [Pixal3D-ComfyUI](https://github.com/Saganaki22/Pixal3D-ComfyUI) — ComfyUI integration with deployment guides for Windows, WSL, and more.
+- [image-to-3dlab (AssetFurnace)](https://github.com/Bingeljell/image-to-3dlab) — an end-to-end Apple Silicon / NVIDIA pipeline that uses Pixal3D as its image-to-3D engine, including a Pixal3D C++/GGML port ([pixal3d.cpp](https://github.com/raven38/pixal3d.cpp)).
 
 ## 🤗 Acknowledgements
 
@@ -299,4 +169,4 @@ If you find this work useful, please consider citing:
 
 ## 📜 License
 
-This project is released under the [MIT License](LICENSE). The third-party components included in this project remain licensed under their respective original terms; see [NOTICE](NOTICE) for the full list of dependencies and their licenses.
+This project is released under the [MIT License](LICENSE). The third-party components included in this project remain licensed under their respective original terms; see [NOTICE](NOTICE) for the full list of dependencies and their licenses.