Update README.md
Browse files
README.md
CHANGED
|
@@ -1,155 +1,167 @@
|
|
| 1 |
# github: https://github.com/Asher-1/InstantMesh-GGML
|
| 2 |
|
| 3 |
-
# MODEL_CARD — InstantMesh GGUF
|
| 4 |
|
| 5 |
-
|
| 6 |
-
|
|
|
|
| 7 |
|
| 8 |
-
**
|
| 9 |
|
| 10 |
-
|
| 11 |
|
| 12 |
```
|
| 13 |
-
|
| 14 |
-
→ ③ synthesizer
|
| 15 |
-
→ ④ flexicubes
|
| 16 |
```
|
| 17 |
|
| 18 |
-
>
|
| 19 |
-
> Ryzen 5950X
|
|
|
|
| 20 |
|
| 21 |
---
|
| 22 |
|
| 23 |
-
## 1.
|
| 24 |
|
| 25 |
```bash
|
| 26 |
-
#
|
| 27 |
huggingface-cli download Asher-1/InstantMeshGGuf --local-dir cpp_ggml/models/gguf
|
| 28 |
|
| 29 |
-
#
|
| 30 |
for f in dino_f16 lrm_transformer_f16 synthesizer_f16; do
|
| 31 |
curl -L -o cpp_ggml/models/gguf/$f.gguf \
|
| 32 |
https://huggingface.co/Asher-1/InstantMeshGGuf/resolve/main/$f.gguf
|
| 33 |
done
|
| 34 |
```
|
| 35 |
|
| 36 |
-
-
|
| 37 |
-
synthesizer_f16`
|
| 38 |
-
- `.gguf`
|
| 39 |
-
|
| 40 |
-
|
| 41 |
|
| 42 |
---
|
| 43 |
|
| 44 |
-
## 2.
|
| 45 |
|
| 46 |
-
|
|
| 47 |
|---|---|---|---|---|
|
| 48 |
-
| **dino** | `dino_*.gguf` | DINOv2
|
| 49 |
-
| **lrm_transformer** | `lrm_transformer_*.gguf` |
|
| 50 |
-
| **synthesizer** | `synthesizer_*.gguf` | OSGDecoder
|
| 51 |
-
|
| 52 |
-
**
|
| 53 |
-
|
| 54 |
-
- **dino**
|
| 55 |
-
token
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
|
|
|
|
|
|
| 65 |
|
| 66 |
---
|
| 67 |
|
| 68 |
-
## 3.
|
| 69 |
|
| 70 |
-
|
|
| 71 |
|---|---|---|---|---|---|
|
| 72 |
| dino | 459.0 | 230.1 | 123.4 | **0.50x** | **0.27x** |
|
| 73 |
| lrm_transformer | 1054.8 | 527.8 | 284.0 | **0.50x** | **0.27x** |
|
| 74 |
| synthesizer | 0.8 | 0.4 | 0.3 | **0.50x** | **0.32x** |
|
| 75 |
-
| **
|
| 76 |
|
| 77 |
-
- f16
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
|
|
|
|
|
|
| 81 |
|
| 82 |
---
|
| 83 |
|
| 84 |
-
## 4.
|
| 85 |
|
| 86 |
-
|
|
| 87 |
|---|---|---|---|---|
|
| 88 |
-
| **CUDA**
|
| 89 |
-
| **Vulkan** | 4.0 | 3.5 | 3.4 |
|
| 90 |
-
| **CPU**
|
| 91 |
-
| PyTorch CUDA
|
| 92 |
-
|
| 93 |
-
-
|
| 94 |
-
- **CPU
|
| 95 |
-
f32
|
| 96 |
-
-
|
| 97 |
-
`../benchmarks/results/perf_ggml_stage_breakdown.png`
|
| 98 |
-
|
| 99 |
|
| 100 |
---
|
| 101 |
|
| 102 |
-
## 5.
|
| 103 |
|
| 104 |
-
### 5.1
|
| 105 |
|
| 106 |
-
|
|
| 107 |
|---|---|---|
|
| 108 |
| CPU | 0.18% | 2.86% |
|
| 109 |
| CUDA | 0.81% | 2.83% |
|
| 110 |
| Vulkan | 0.76% | 2.00% |
|
| 111 |
|
| 112 |
-
### 5.2
|
| 113 |
|
| 114 |
-
|
|
| 115 |
|---|---|
|
| 116 |
| CPU | 0.64% |
|
| 117 |
| Vulkan | 0.72% |
|
| 118 |
|
| 119 |
-
- **f16
|
| 120 |
-
SDF
|
| 121 |
-
|
|
|
|
|
|
|
| 122 |
|
| 123 |
---
|
| 124 |
|
| 125 |
-
## 6.
|
| 126 |
|
| 127 |
-
|
|
| 128 |
|---|---|---|
|
| 129 |
-
|
|
| 130 |
-
|
|
| 131 |
-
| AMD / Intel /
|
| 132 |
-
|
|
| 133 |
-
|
|
| 134 |
-
|
|
| 135 |
-
|
|
| 136 |
-
|
| 137 |
-
**
|
| 138 |
-
|
| 139 |
-
- **dino**
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
|
|
|
|
|
|
| 145 |
|
| 146 |
---
|
| 147 |
|
| 148 |
-
## 7.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
|
| 150 |
-
- 跑批脚本:`bash ../benchmarks/run_bench.sh`(3 后端 × 3 精度 × 4 demo,
|
| 151 |
-
生成 `times.csv` 与各 run 的 `.sdf.bin` / `.obj`)
|
| 152 |
-
- 分析/绘图:`python3 ../benchmarks/analyze.py`(生成 `report.md`、
|
| 153 |
-
`latency.png`、`precision_rmse.png`、`precision_maxerr.png`)
|
| 154 |
-
- 权重转换(官方 ckpt → GGUF,三精度):`python3 -m convert.convert_all`
|
| 155 |
-
- 手动推理:见本模块根 [README.md §3 Step 5](../README.md) 的完整命令。
|
|
|
|
| 1 |
# github: https://github.com/Asher-1/InstantMesh-GGML
|
| 2 |
|
| 3 |
+
# MODEL_CARD — InstantMesh GGUF Inference Model Card
|
| 4 |
|
| 5 |
+
This directory (`cpp_ggml/models/gguf/`) holds the 3 ggml models required by the
|
| 6 |
+
InstantMesh end-to-end reconstruction pipeline, each provided as `.gguf` weights in
|
| 7 |
+
three precisions: `f32 / f16 / q8`.
|
| 8 |
|
| 9 |
+
**Weight download (Hugging Face): https://huggingface.co/Asher-1/InstantMeshGGuf**
|
| 10 |
|
| 11 |
+
Pipeline and data flow:
|
| 12 |
|
| 13 |
```
|
| 14 |
+
Image → ① dino (image encoding) → ② lrm_transformer (triplane generation)
|
| 15 |
+
→ ③ synthesizer (OSGDecoder voxel decoding: SDF/deform/weight/RGB)
|
| 16 |
+
→ ④ flexicubes (SDF → mesh, no weights)
|
| 17 |
```
|
| 18 |
|
| 19 |
+
> Performance and accuracy data in this card come from `../benchmarks/results/report.md`
|
| 20 |
+
> (RTX 3060 / Ryzen 5950X, grid_res=88, same conditions as the PyTorch reference;
|
| 21 |
+
> end-to-end includes model loading and mesh export).
|
| 22 |
|
| 23 |
---
|
| 24 |
|
| 25 |
+
## 1. Download and Usage
|
| 26 |
|
| 27 |
```bash
|
| 28 |
+
# All 9 files (~2.7 GB)
|
| 29 |
huggingface-cli download Asher-1/InstantMeshGGuf --local-dir cpp_ggml/models/gguf
|
| 30 |
|
| 31 |
+
# Or the minimal usable set: the f16 trio (~760 MB)
|
| 32 |
for f in dino_f16 lrm_transformer_f16 synthesizer_f16; do
|
| 33 |
curl -L -o cpp_ggml/models/gguf/$f.gguf \
|
| 34 |
https://huggingface.co/Asher-1/InstantMeshGGuf/resolve/main/$f.gguf
|
| 35 |
done
|
| 36 |
```
|
| 37 |
|
| 38 |
+
- The three models **must use the same precision** (`dino_f16 + lrm_transformer_f16 +
|
| 39 |
+
synthesizer_f16`); mixing precisions may cause inter-layer numeric mismatches.
|
| 40 |
+
- `.gguf` files are not committed (large binaries); the weights contain network
|
| 41 |
+
parameters only, **no preprocessing parameters**. Inputs must be ImageNet-normalized
|
| 42 |
+
per DINOv2 rules (see `../src/tools/instantmesh.cpp` and `../convert/prep_input.py`).
|
| 43 |
|
| 44 |
---
|
| 45 |
|
| 46 |
+
## 2. Model Overview
|
| 47 |
|
| 48 |
+
| Model | File prefix | Role | Input | Output |
|
| 49 |
|---|---|---|---|---|
|
| 50 |
+
| **dino** | `dino_*.gguf` | DINOv2 vision Transformer backbone extracting multi-scale patch features | Multi-view RGB tensor [V,3,224,224] (ImageNet-normalized) | Image token sequence |
|
| 51 |
+
| **lrm_transformer** | `lrm_transformer_*.gguf` | Reconstruction backbone fusing image features into triplane latents | Image token sequence | triplane [3, C, H, W] |
|
| 52 |
+
| **synthesizer** | `synthesizer_*.gguf` | OSGDecoder: samples the triplane at voxel points and decodes | triplane + sample point coordinates | SDF / deformation / FlexiCubes weight / RGB |
|
| 53 |
+
|
| 54 |
+
**Model details**
|
| 55 |
+
|
| 56 |
+
- **dino**: splits the input image into patches and encodes them with self-attention
|
| 57 |
+
(flash attention), outputting a token sequence. It determines "how much detail the
|
| 58 |
+
model can see in the image" and is the feature source for all subsequent reconstruction.
|
| 59 |
+
- **lrm_transformer**: 16 layers of 1024-dim BasicTransformerBlock (self/cross-attention +
|
| 60 |
+
GELU MLP + adaLN modulation) projecting DINO features into three orthogonal feature
|
| 61 |
+
planes (triplane), implicitly encoding the full object in a compact low-dimensional space.
|
| 62 |
+
- **synthesizer**: given sample points, projects them onto the triplanes, concatenates
|
| 63 |
+
the features and decodes via a small MLP the point's **SDF** (signed distance),
|
| 64 |
+
**deformation** (vertex deformation), **weight** (FlexiCubes weights) and **RGB**
|
| 65 |
+
(`net_rgb` branch, shared by vertex colors and texture baking).
|
| 66 |
+
|
| 67 |
+
The three are chained: without dino there are no image features, without
|
| 68 |
+
lrm_transformer no triplane, without synthesizer no SDF/mesh/colors.
|
| 69 |
|
| 70 |
---
|
| 71 |
|
| 72 |
+
## 3. File Size and Memory Footprint
|
| 73 |
|
| 74 |
+
| Model | f32 (MB) | f16 (MB) | q8 (MB) | f16/f32 | q8/f32 |
|
| 75 |
|---|---|---|---|---|---|
|
| 76 |
| dino | 459.0 | 230.1 | 123.4 | **0.50x** | **0.27x** |
|
| 77 |
| lrm_transformer | 1054.8 | 527.8 | 284.0 | **0.50x** | **0.27x** |
|
| 78 |
| synthesizer | 0.8 | 0.4 | 0.3 | **0.50x** | **0.32x** |
|
| 79 |
+
| **All three models** | **1514.6** | **758.3** | **407.7** | 0.50x | 0.27x |
|
| 80 |
|
| 81 |
+
- f16 is always half of f32 and q8 about a quarter, matching the theoretical
|
| 82 |
+
quantization size ratios.
|
| 83 |
+
- Size is **dominated by lrm_transformer** (~1 GB at f32): it drives most of the
|
| 84 |
+
end-to-end memory/load time. Measured ggml CUDA f16 end-to-end peak memory is
|
| 85 |
+
**1011 MB** (PyTorch under the same conditions: 3725 MB); an RTX 3060 with 12 GB
|
| 86 |
+
VRAM runs comfortably at grid_res=88.
|
| 87 |
|
| 88 |
---
|
| 89 |
|
| 90 |
+
## 4. Performance (end-to-end, seconds per image)
|
| 91 |
|
| 92 |
+
| Backend | f32 | f16 | q8 | Notes |
|
| 93 |
|---|---|---|---|---|
|
| 94 |
+
| **CUDA** (RTX 3060) | 3.5 | **3.0** | **2.9** | Recommended; **4.5x** faster than PyTorch CUDA (13.5s f16) |
|
| 95 |
+
| **Vulkan** | 4.0 | 3.5 | 3.4 | Preferred path for cross-vendor GPUs (AMD/Intel) |
|
| 96 |
+
| **CPU** (Ryzen 5950X) | 24.8 | 24.4 | 23.5 | Usable without a GPU; **avoid f16** |
|
| 97 |
+
| PyTorch CUDA (official) | 13.7 | 13.5 | — | Reference baseline |
|
| 98 |
+
|
| 99 |
+
- On GPUs, **q8 is the fastest**, f16 next; CUDA is ~10–15% faster than Vulkan.
|
| 100 |
+
- **On CPU, f16 is actually the slowest/unstable**: caused by the CPU-side f16→f32
|
| 101 |
+
dequantization overhead; choose f32 or q8 on CPU.
|
| 102 |
+
- Per-stage timing breakdown (ggml CUDA f16): see
|
| 103 |
+
`../benchmarks/results/perf_ggml_stage_breakdown.png`; lrm_transformer dominates,
|
| 104 |
+
dino is next, and synthesizer takes a tiny share.
|
| 105 |
|
| 106 |
---
|
| 107 |
|
| 108 |
+
## 5. Accuracy (SDF relative error, mean|sdf|≈8.5, grid_res=88)
|
| 109 |
|
| 110 |
+
### 5.1 Pure quantization error (same backend, vs each backend's own f32)
|
| 111 |
|
| 112 |
+
| Backend | f16 relative RMSE | q8 relative RMSE |
|
| 113 |
|---|---|---|
|
| 114 |
| CPU | 0.18% | 2.86% |
|
| 115 |
| CUDA | 0.81% | 2.83% |
|
| 116 |
| Vulkan | 0.76% | 2.00% |
|
| 117 |
|
| 118 |
+
### 5.2 Backend consistency (f32 vs CUDA-f32)
|
| 119 |
|
| 120 |
+
| Backend | Relative RMSE |
|
| 121 |
|---|---|
|
| 122 |
| CPU | 0.64% |
|
| 123 |
| Vulkan | 0.72% |
|
| 124 |
|
| 125 |
+
- **f16 is nearly lossless** (relative error <1%), mesh geometry unchanged; **q8 is
|
| 126 |
+
acceptable** (2–3%), SDF sign-flip rate <0.8%, geometry essentially indistinguishable
|
| 127 |
+
to the eye.
|
| 128 |
+
- **Across the three backends, same-precision results agree within <1%**: quantization
|
| 129 |
+
and backend implementations are consistent — no "backend silently losing accuracy".
|
| 130 |
|
| 131 |
---
|
| 132 |
|
| 133 |
+
## 6. Recommended Use Cases
|
| 134 |
|
| 135 |
+
| Scenario | Recommendation | Rationale |
|
| 136 |
|---|---|---|
|
| 137 |
+
| Everyday inference on NVIDIA GPUs | **CUDA + f16** | <1% accuracy loss, 3.0 s/image, ~1 GB peak memory |
|
| 138 |
+
| Maximum speed / batch throughput | CUDA + **q8** | Fastest (2.9 s), smallest size (408 MB for the trio), 2–3% accuracy acceptable |
|
| 139 |
+
| AMD / Intel / GPUs without CUDA drivers | **Vulkan + f16** | The only cross-vendor GPU path, 3.5 s/image |
|
| 140 |
+
| Tight VRAM (≤4 GB) or mobile | **q8** (lrm_transformer only 284 MB) | Smallest peak memory |
|
| 141 |
+
| Server CPU deployment | CPU + **f32 or q8** | Avoids the slow CPU f16 dequantization pitfall |
|
| 142 |
+
| Accuracy benchmarking / regression tests / research reproduction | **f32** | Layer-by-layer comparable with the PyTorch reference; serves as the parity reference |
|
| 143 |
+
| Validating the pipeline on edge devices | **q8 CPU** | No GPU dependency at all, runs at ~24 s/image |
|
| 144 |
+
|
| 145 |
+
**Per-model advice**
|
| 146 |
+
|
| 147 |
+
- **dino**: determines feature quality; generally **not lower than f16**. Use q8 only
|
| 148 |
+
for maximum speed when a slight quality loss is acceptable. Input resolution is
|
| 149 |
+
fixed at 224×224.
|
| 150 |
+
- **lrm_transformer**: the largest model, where quantization gains and risks are most
|
| 151 |
+
concentrated; if q8 shows geometric distortion, fall back to f16.
|
| 152 |
+
- **synthesizer**: <1 MB, quantization gains are negligible; **prefer f16 or f32
|
| 153 |
+
directly** to avoid pointless accuracy loss; its `plane_dim` must match the
|
| 154 |
+
lrm_transformer output.
|
| 155 |
|
| 156 |
---
|
| 157 |
|
| 158 |
+
## 7. Reproduction and Extension
|
| 159 |
+
|
| 160 |
+
- Batch runner: `bash ../benchmarks/run_bench.sh` (3 backends × 3 precisions × 4 demos,
|
| 161 |
+
producing `times.csv` and per-run `.sdf.bin` / `.obj`)
|
| 162 |
+
- Analysis/plots: `python3 ../benchmarks/analyze.py` (generates `report.md`,
|
| 163 |
+
`latency.png`, `precision_rmse.png`, `precision_maxerr.png`)
|
| 164 |
+
- Weight conversion (official ckpt → GGUF, three precisions): `python3 -m convert.convert_all`
|
| 165 |
+
- Manual inference: see the full commands in [README.md §3 Step 5](../README.md) at the
|
| 166 |
+
module root.
|
| 167 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|