| { | |
| "name": "Pixal3D (proj, single-image; experimental MV mean-pool)", | |
| "dir": "pixal3d", | |
| "run": "run_infer.sh", | |
| "min_images": 1, | |
| "max_images": 4, | |
| "single_image": true, | |
| "notes": "TencentARC/Pixal3D (SIGGRAPH 2026) — pixel-aligned image-to-3D built on TRELLIS.2 / Direct3D-S2. Produces a TEXTURED GLB with PBR attributes (base_color 4096x4096 + metallic/roughness) via a 3-stage cascade: sparse-structure DiT (32/64) -> shape SLat DiT (512->1024/1536) -> texture SLat DiT, decoded to mesh + voxel PBR by o_voxel.postprocess.to_glb. Conditioning is 'proj' (view-aligned back-projection of DINOv3 patch features + NAF hi-res upsampling) with a MoGe-2-estimated camera (angle_x + distance) per image. IMPORTANT: the released proj weights + inference are architecturally SINGLE-IMAGE — inference.py/app.py/pipeline.run all hardcode batch 1, and ProjGrid.forward asserts transform_matrix is None (every view is projected through a FIXED canonical front-view camera; per-view pose is never used). There is NO native multi-view inference entrypoint or fusion in the released code; the only true multi-view fusion (token-concat) is orphaned non-proj training code with no checkpoint. run_infer.sh therefore produces two GLBs per sample: <s>_seed42_singleview.glb (stock single-image on the canonical front view — the FAITHFUL output) and <s>_seed42.glb (EXPERIMENTAL multi-view via inference_mv.py, which mean-pools the DINOv3 proj-grid + global features across V views before the sparse-coords indexing). The MV variant is OOD (each view forced through the front camera, DiT trained on 1 view) and is provided for comparison, not as validated multi-view. Env: conda prefix /lp-dev/jonghoon/mv-mesh/envs/pixal3d, python3.10 + torch2.6.0+cu124, ATTN_BACKEND=sdpa (flash_attn_3 wheel is Hopper-only; A100 uses SDPA). natten 0.21.0 was REBUILT from source for sm_80 (the shipped LDYang694 wheel is sm_90-only); flex_gemm/cumesh/o_voxel/nvdiffrast wheels already include sm_80. Background removal (gated briaai/RMBG-2.0) is bypassed by feeding mask alpha (mv_common.load_rgba) with a lazy non-gated ZhengPeng7/BiRefNet fallback. Weights ~23GB under hf_cache (TencentARC/Pixal3D) + DINOv3-vitl (camenduru mirror) + MoGe-2-vitl + NAF. Inputs: ordered --images with aligned --masks; mean-pool is order-agnostic. Seed 42.", | |
| "status": "ready" | |
| } | |