model card

README.md

@@ -1,3 +1,181 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+pipeline_tag: image-to-3d
+tags:
+  - localmesh-engine
+  - image-to-3d
+  - mesh
+  - gltf
+  - trellis
+  - multi-view
+  - 3d-generation
+base_model: TencentARC/Pixal3D
+base_model_relation: quantized
+---
+
+![LocalMesh Engine](https://raw.githubusercontent.com/Quentincls/localmesh-engine/main/assets/banner.png)
+
+# LocalMesh Engine, multi-view weights
+
+LocalMesh Engine turns one photo, or four sides of the same subject, into a
+textured `.glb`. It runs on an 8 GB NVIDIA card. The code is Apache-2.0 and
+lives on [GitHub](https://github.com/Quentincls/localmesh-engine). The project
+page is [local-mesh.com/localmesh-engine](https://local-mesh.com/localmesh-engine/).
+
+TRELLIS.2 works from a single image. This repository holds the weights of the
+four view path: the three fp8 conversions of the Pixal3D multi-view models,
+and the field network that recovers detail between image tokens. The three
+conversions exist nowhere else. Everything else the engine needs comes from
+upstream repositories and is listed below.
+
+## What is in this repository
+
+| File | Size | What it does | Source |
+|---|---|---|---|
+| `structure_mv_fp8.safetensors` + `.json` | 1.39 GB | Fuses the four encoded views into a sparse volume of cells. Each view is projected onto the shared grid by its own camera. | fp8 conversion of `ckpts/ss_flow_img_dit_1_3B_64_bf16_mv.safetensors`, TencentARC/Pixal3D |
+| `forme_512_mv_fp8.safetensors` + `.json` | 1.44 GB | First shape pass, on the grid inherited from the structure. | fp8 conversion of `ckpts/slat_flow_img2shape_dit_1_3B_512_bf16_mv.safetensors`, TencentARC/Pixal3D |
+| `forme_1024_mv_fp8.safetensors` + `.json` | 1.44 GB | Second shape pass. The TRELLIS.2 shape decoder turns its latent into the mesh. | fp8 conversion of `ckpts/slat_flow_img2shape_dit_1_3B_1024_bf16_mv.safetensors`, TencentARC/Pixal3D |
+| `champ.safetensors` | 2.7 MB | NAF field network. Returns a 512 by 512 query map, which fills the detail the token map loses: the token map is sixteen times coarser than the photo. | valeoai/NAF, official checkpoint, tensors unchanged |
+
+4.28 GB in total, seven files. Keep the three `.json` descriptors next to their
+`.safetensors`: the engine builds each flow model on the `meta` device from that
+descriptor, then loads the tensors in place, so the weights are never held
+twice. The descriptors declare `dtype: float8_e4m3fn`.
+
+The four view path serves the `draft`, `standard` and `high` tiers. The
+multi-view weights exist at 512 and 1024 only, so `max` falls back to the
+earlier way of blending the views rather than shipping a `standard` shape under
+another name.
+
+## What is not in this repository
+
+| Weights | Where | Licence | Note |
+|---|---|---|---|
+| TRELLIS.2-4B, fp8 | https://huggingface.co/visualbruno/TRELLIS.2-4B-FP8 | MIT | 8.1 GB. Texture on both paths, the single photo path, and the shape decoder the four view path ends on. Required either way. |
+| TRELLIS-image-large, structure decoder | https://huggingface.co/microsoft/TRELLIS-image-large | MIT | Two files, `ss_dec_conv3d_16l8_fp16.json` and `.safetensors`. |
+| DINOv3 ViT-L/16 | https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m | DINOv3 License, Meta | Image encoder, required on both paths. Access is gated and approved by hand, so ask for it first. |
+| BiRefNet_HR | https://huggingface.co/ZhengPeng7/BiRefNet_HR | MIT | Cutout. The engine fetches this one from the Hub on the first generation if the cache is empty, and loads it with `trust_remote_code=True`, so that first run executes code from the Hub. |
+| DA3-BASE | https://huggingface.co/depth-anything/DA3-BASE | Apache-2.0 | `model.safetensors` (541 MB) and `config.json`, plus the `depth_anything_3` source tree beside them: 544 MB in place. Measures the azimuth of each shot and which side each profile shows. The engine runs without it, but it then guesses which side each profile is on, and says so in its result. A wrong guess puts a face at the front and at the back. |
+
+## Using them
+
+Set `LOCALMESH_ROOT` to the folder that holds `models/`, then:
+
+```bash
+pip install -U huggingface_hub
+hf download Qtn-Cls/LocalMeshEngine --local-dir "$LOCALMESH_ROOT/models/multivue"
+```
+
+This repository is flat, so the seven files land exactly where the engine looks
+for them. There is nothing to move afterwards.
+
+The layout the engine reads:
+
+```
+<LOCALMESH_ROOT>/models/
+  TRELLIS.2-4B/                              pipeline.json (or pipeline_fp8.json), ckpts_fp8/
+  microsoft/TRELLIS-image-large/ckpts/       ss_dec_conv3d_16l8_fp16.json + .safetensors
+  facebook/dinov3-vitl16-pretrain-lvd1689m/
+  multivue/                                  this repository
+  multivue/cameras/                          DA3-BASE, model.safetensors + config.json
+  hf/                                        Hugging Face cache (HF_HOME), where BiRefNet_HR lands
+```
+
+Then, from the four sides of one subject:
+
+```bash
+python -m localmesh_engine face.png --droite right.png --gauche left.png --dos back.png --vers out/
+```
+
+`--palier` picks the tier: `draft`, `standard`, `high` or `max`, written
+Draft, Standard, Detailed and Extreme where these pages spell them out.
+`--graine` sets the seed, `--vers` the output folder. The command assumes the package is installed.
+The four view path also needs `natten`. Installation, including the three CUDA
+extensions that are not on PyPI, is written up in `docs/INSTALL.md` in the
+GitHub repository; the four tiers, frozen, are in `docs/RECIPES.md`.
+
+## Results
+
+![Four views to one .glb](https://raw.githubusercontent.com/Quentincls/localmesh-engine/main/assets/four-views.png)
+
+![Gallery](https://raw.githubusercontent.com/Quentincls/localmesh-engine/main/assets/gallery.png)
+
+![Turntable](https://raw.githubusercontent.com/Quentincls/localmesh-engine/main/assets/turntable.gif)
+
+Six subjects, four photos each, `high` tier, RTX 4060 Laptop 8 GB: samurai on a
+base, crowned stone head, sword in the stone, motorcycle, cassette with a clear
+shell, traffic light.
+
+| Tier | Four views, measured over the six subjects |
+|---|---|
+| Standard, `standard` | 6 min 30 to 8 min 30 |
+| Detailed, `high` | 9 min 20 to 13 min |
+
+Texture accounts for about 60 % of that time.
+
+## The conversion
+
+The three flow models start from the official `*_mv` weights of
+TencentARC/Pixal3D. Their tensors are stored in float32, despite the `bf16` in
+their filenames. The transformer blocks are cast to `float8_e4m3fn`, and the
+descriptor beside each file records that dtype, so the engine builds the model
+in fp8 rather than casting after the fact. Of each model, 480 tensors are
+converted; the input and output layers, the norms and modulations, and the
+structure model's complex rotary table are left as they were.
+
+Measured against the source: RMSE of 0.025 to 0.026 on the weights, and 0.027
+on a projection probe.
+
+`champ.safetensors` is not converted. It carries the tensors of the official
+valeoai/NAF checkpoint, `naf_release.pth`, unchanged, re-serialised to
+safetensors. The file records the source URL and its SHA-256 in its own
+metadata.
+
+## Licences and attribution
+
+The `license` field above is the repository tag. It is Apache-2.0, the licence
+of the engine code and of this card. The files themselves keep the licence of
+their own source:
+
+- The three fp8 conversions derive from **TencentARC/Pixal3D**, MIT License,
+  Copyright (c) 2026 Tencent. Code: https://github.com/TencentARC/Pixal3D,
+  weights: https://huggingface.co/TencentARC/Pixal3D
+- `champ.safetensors` comes from **valeoai/NAF**, Apache-2.0,
+  https://github.com/valeoai/NAF
+
+The engine also builds on:
+
+- **TRELLIS.2**, Microsoft, MIT License, https://github.com/microsoft/TRELLIS.2
+- **ComfyUI-Trellis2**, visualbruno, MIT License, for the ported multi-view path
+  and the fp8 loading, https://github.com/visualbruno/ComfyUI-Trellis2
+- **Depth Anything 3**, Apache-2.0, for the camera measurement,
+  https://github.com/ByteDance-Seed/depth-anything-3
+- **BiRefNet**, MIT License, for the cutout,
+  https://github.com/ZhengPeng7/BiRefNet
+
+Built with DINOv3. DINOv3 is the image encoder on both paths. Its weights are
+not redistributed here: request them from Meta, and ship a copy of the DINOv3
+License Agreement with any redistribution of your own.
+
+## Citing
+
+```bibtex
+@software{colus2026localmeshengine,
+  author  = {Colus, Quentin},
+  title   = {LocalMesh Engine},
+  year    = {2026},
+  version = {1.0.0},
+  license = {Apache-2.0},
+  url     = {https://github.com/Quentincls/localmesh-engine}
+}
+```
+
+The upstream work these weights rest on: TRELLIS.2 (Microsoft,
+arXiv:2512.14692), Pixal3D (TencentARC, arXiv:2605.10922), NAF (valeoai),
+DINOv3 (Meta), Depth Anything 3 (arXiv:2511.10647).
+
+## Code
+
+The engine is at **https://github.com/Quentincls/localmesh-engine**. Recipes,
+installation, measurements and the provenance of every vendored file are there.
+Open issues there, not here.