Asher-1 commited on
Commit
ae15501
·
verified ·
1 Parent(s): 485a137

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +102 -90
README.md CHANGED
@@ -1,155 +1,167 @@
1
  # github: https://github.com/Asher-1/InstantMesh-GGML
2
 
3
- # MODEL_CARD — InstantMesh GGUF 推理模型卡
4
 
5
- 本目录(`cpp_ggml/models/gguf/`)存放 InstantMesh 端到端重建管线所需的
6
- 3 个 ggml 模型,均提供 `f32 / f16 / q8` 三种精度的 `.gguf` 权重。
 
7
 
8
- **权重下载(Hugging Face):https://huggingface.co/Asher-1/InstantMeshGGuf**
9
 
10
- 管线与数据流:
11
 
12
  ```
13
- 图片 → ① dino(图像编码) → ② lrm_transformer(三平面生成)
14
- → ③ synthesizer(OSGDecoder 体素解码:SDF/deform/weight/RGB)
15
- → ④ flexicubes(SDF → mesh,无权重)
16
  ```
17
 
18
- > 本文性能与精度数据来自 `../benchmarks/results/report.md`(RTX 3060 /
19
- > Ryzen 5950X,grid_res=88,与 PyTorch 参考同条件;端到端含模型加载与网格导出)。
 
20
 
21
  ---
22
 
23
- ## 1. 下载与配套使用
24
 
25
  ```bash
26
- # 全部 9 个文件(约 2.7 GB)
27
  huggingface-cli download Asher-1/InstantMeshGGuf --local-dir cpp_ggml/models/gguf
28
 
29
- # 或最小可用集:f16 三件套(约 760 MB)
30
  for f in dino_f16 lrm_transformer_f16 synthesizer_f16; do
31
  curl -L -o cpp_ggml/models/gguf/$f.gguf \
32
  https://huggingface.co/Asher-1/InstantMeshGGuf/resolve/main/$f.gguf
33
  done
34
  ```
35
 
36
- - 三个模型**必须配套同一精度**(`dino_f16 + lrm_transformer_f16 +
37
- synthesizer_f16`),混合精度可能导致层间数值不匹配。
38
- - `.gguf` 不入库(大二进制);权重只含网络参数,**不含预处理参数**,
39
- 输入需按 DINOv2 规则做 ImageNet 归一化(见 `../src/tools/instantmesh.cpp`
40
- 与 `../convert/prep_input.py`)。
41
 
42
  ---
43
 
44
- ## 2. 模型总览
45
 
46
- | 模型 | 文件前缀 | 作用 | 输入 | 输出 |
47
  |---|---|---|---|---|
48
- | **dino** | `dino_*.gguf` | DINOv2 视觉 Transformer 主干,提取多尺度 patch 特征 | 多视角 RGB 张量 [V,3,224,224](ImageNet 归一化) | 图像 token 序列 |
49
- | **lrm_transformer** | `lrm_transformer_*.gguf` | 重建主干,把图像特征融合为三平面潜变量 | 图像 token 序列 | triplane [3, C, H, W] |
50
- | **synthesizer** | `synthesizer_*.gguf` | OSGDecoder:体素点上采样 triplane 并解码 | triplane + 采样点坐标 | SDF / deformation / FlexiCubes weight / RGB |
51
-
52
- **各模型详解**
53
-
54
- - **dino**:把输入图切成 patch 后做自注意力编码(flash attention),输出
55
- token 序列。决定"模型能从图像里看到多少细节",是所有后续重建的特征来源。
56
- - **lrm_transformer**:16 层 1024 维 BasicTransformerBlock(自/交叉注意力 +
57
- GELU MLP + adaLN 调制),把 DINO 特征投影为三个正交特征平面(triplane),
58
- 以紧凑低维空间隐式编码完整物体。
59
- - **synthesizer**:给定采样点,投影到三平面取特征拼接后经小 MLP 解码出该点
60
- 的 **SDF**(有符号距离)、**deformation**(顶点形变)、**weight**(FlexiCubes
61
- 权重)与 **RGB**(`net_rgb` 分支,顶点色/纹理烘焙共用)。
62
-
63
- 三者串联:没有 dino 就没有图像特征,没有 lrm_transformer 就没有三平面,
64
- 没有 synthesizer 就没有 SDF/mesh/颜色。
 
 
65
 
66
  ---
67
 
68
- ## 3. 体积与显存占用
69
 
70
- | 模型 | f32 (MB) | f16 (MB) | q8 (MB) | f16/f32 | q8/f32 |
71
  |---|---|---|---|---|---|
72
  | dino | 459.0 | 230.1 | 123.4 | **0.50x** | **0.27x** |
73
  | lrm_transformer | 1054.8 | 527.8 | 284.0 | **0.50x** | **0.27x** |
74
  | synthesizer | 0.8 | 0.4 | 0.3 | **0.50x** | **0.32x** |
75
- | **三模型合计** | **1514.6** | **758.3** | **407.7** | 0.50x | 0.27x |
76
 
77
- - f16 恒为 f32 一半、q8 约四分之一,符合量化理论体积比。
78
- - 体量由 **lrm_transformer 主导**(约 1 GB f32):端到端显存/加载耗时主要
79
- 来自它。ggml CUDA f16 端到端峰值内存实测 **1011 MB**(PyTorch 同条件
80
- 3725 MB),12 GB 显存的 RTX 3060 可在 grid_res=88 下从容运行。
 
 
81
 
82
  ---
83
 
84
- ## 4. 性能指标(端到端,秒/图)
85
 
86
- | 后端 | f32 | f16 | q8 | 备注 |
87
  |---|---|---|---|---|
88
- | **CUDA**(RTX 3060) | 3.5 | **3.0** | **2.9** | 推荐;比 PyTorch CUDA(13.5s f16)快 **4.5x** |
89
- | **Vulkan** | 4.0 | 3.5 | 3.4 | 跨厂商 GPU(AMD/Intel)首选 |
90
- | **CPU**(Ryzen 5950X) | 24.8 | 24.4 | 23.5 | 无 GPU 时可用;**避免 f16** |
91
- | PyTorch CUDA(官方) | 13.7 | 13.5 | — | 对照基线 |
92
-
93
- - GPU 上 **q8 最快**、f16 次之;CUDA 比 Vulkan 快约 10–15%。
94
- - **CPU 上 f16 反而最慢/不稳定**:CPU 端 f16→f32 反量化开销所致,CPU 请选
95
- f32 或 q8。
96
- - 阶段耗时拆分(ggml CUDA f16):见
97
- `../benchmarks/results/perf_ggml_stage_breakdown.png`,大头为
98
- lrm_transformer,dino 次之,synthesizer 占比极小。
99
 
100
  ---
101
 
102
- ## 5. 精度指标(SDF 相对误差,mean|sdf|≈8.5,grid_res=88)
103
 
104
- ### 5.1 纯量化误差(同后端,相对各自 f32)
105
 
106
- | 后端 | f16 相对RMSE | q8 相对RMSE |
107
  |---|---|---|
108
  | CPU | 0.18% | 2.86% |
109
  | CUDA | 0.81% | 2.83% |
110
  | Vulkan | 0.76% | 2.00% |
111
 
112
- ### 5.2 后端一致性(f32 相对 CUDA-f32)
113
 
114
- | 后端 | 相对RMSE |
115
  |---|---|
116
  | CPU | 0.64% |
117
  | Vulkan | 0.72% |
118
 
119
- - **f16 几乎无损**(相对误差 <1%),网格几何不变;**q8 可接受**(2–3%),
120
- SDF 符号翻转率 <0.8%,肉眼几何基本无差。
121
- - **三后端同精度互差 <1%**:量化与后端实现一致,无"某后端偷偷掉精度"。
 
 
122
 
123
  ---
124
 
125
- ## 6. 适用场景推荐
126
 
127
- | 场景 | 推荐 | 理由 |
128
  |---|---|---|
129
- | NVIDIA GPU 日常推理 | **CUDA + f16** | 精度损失 <1%、3.0 s/图、峰值内存 ~1 GB |
130
- | 追求极致速度 / 批量吞吐 | CUDA + **q8** | 最快(2.9 s)、体积最小(408 MB 三件套),精度 2–3% 可接受 |
131
- | AMD / Intel / 无 CUDA 驱动 GPU | **Vulkan + f16** | 跨厂商唯一 GPU 路径,3.5 s/图 |
132
- | 显存紧张(≤4 GB)或移动端 | **q8**(lrm_transformer 仅 284 MB) | 峰值内存最小 |
133
- | 服务器 CPU 部署 | CPU + **f32 或 q8** | 避开 CPU f16 反量化慢的坑 |
134
- | 精度基准 / 回归测试 / 科研复现 | **f32** | 与 PyTorch 参考逐层可比,作 parity 参考 |
135
- | 边缘设备先验证流程 | **q8 CPU** | 无任何 GPU 依赖,~24 s/图可跑通 |
136
-
137
- **分模型建议**
138
-
139
- - **dino**:决定特征质量,一般**不低于 f16**;仅极致速度且接受轻微质量
140
- 损失时用 q8。输入分辨率固定 224×224。
141
- - **lrm_transformer**:体量最大,量化收益/风险都最集中;q8 若出现几何畸变
142
- 请回退 f16。
143
- - **synthesizer**:<1 MB,量化收益可忽略,**建议直接 f16 或 f32**,避免无谓
144
- 精度损失;其 `plane_dim` 必须与 lrm_transformer 输出匹配。
 
 
145
 
146
  ---
147
 
148
- ## 7. 复现与扩展
 
 
 
 
 
 
 
 
149
 
150
- - 跑批脚本:`bash ../benchmarks/run_bench.sh`(3 后端 × 3 精度 × 4 demo,
151
- 生成 `times.csv` 与各 run 的 `.sdf.bin` / `.obj`)
152
- - 分析/绘图:`python3 ../benchmarks/analyze.py`(生成 `report.md`、
153
- `latency.png`、`precision_rmse.png`、`precision_maxerr.png`)
154
- - 权重转换(官方 ckpt → GGUF,三精度):`python3 -m convert.convert_all`
155
- - 手动推理:见本模块根 [README.md §3 Step 5](../README.md) 的完整命令。
 
1
  # github: https://github.com/Asher-1/InstantMesh-GGML
2
 
3
+ # MODEL_CARD — InstantMesh GGUF Inference Model Card
4
 
5
+ This directory (`cpp_ggml/models/gguf/`) holds the 3 ggml models required by the
6
+ InstantMesh end-to-end reconstruction pipeline, each provided as `.gguf` weights in
7
+ three precisions: `f32 / f16 / q8`.
8
 
9
+ **Weight download (Hugging Face): https://huggingface.co/Asher-1/InstantMeshGGuf**
10
 
11
+ Pipeline and data flow:
12
 
13
  ```
14
+ Image → ① dino (image encoding) → ② lrm_transformer (triplane generation)
15
+ → ③ synthesizer (OSGDecoder voxel decoding: SDF/deform/weight/RGB)
16
+ → ④ flexicubes (SDF → mesh, no weights)
17
  ```
18
 
19
+ > Performance and accuracy data in this card come from `../benchmarks/results/report.md`
20
+ > (RTX 3060 / Ryzen 5950X, grid_res=88, same conditions as the PyTorch reference;
21
+ > end-to-end includes model loading and mesh export).
22
 
23
  ---
24
 
25
+ ## 1. Download and Usage
26
 
27
  ```bash
28
+ # All 9 files (~2.7 GB)
29
  huggingface-cli download Asher-1/InstantMeshGGuf --local-dir cpp_ggml/models/gguf
30
 
31
+ # Or the minimal usable set: the f16 trio (~760 MB)
32
  for f in dino_f16 lrm_transformer_f16 synthesizer_f16; do
33
  curl -L -o cpp_ggml/models/gguf/$f.gguf \
34
  https://huggingface.co/Asher-1/InstantMeshGGuf/resolve/main/$f.gguf
35
  done
36
  ```
37
 
38
+ - The three models **must use the same precision** (`dino_f16 + lrm_transformer_f16 +
39
+ synthesizer_f16`); mixing precisions may cause inter-layer numeric mismatches.
40
+ - `.gguf` files are not committed (large binaries); the weights contain network
41
+ parameters only, **no preprocessing parameters**. Inputs must be ImageNet-normalized
42
+ per DINOv2 rules (see `../src/tools/instantmesh.cpp` and `../convert/prep_input.py`).
43
 
44
  ---
45
 
46
+ ## 2. Model Overview
47
 
48
+ | Model | File prefix | Role | Input | Output |
49
  |---|---|---|---|---|
50
+ | **dino** | `dino_*.gguf` | DINOv2 vision Transformer backbone extracting multi-scale patch features | Multi-view RGB tensor [V,3,224,224] (ImageNet-normalized) | Image token sequence |
51
+ | **lrm_transformer** | `lrm_transformer_*.gguf` | Reconstruction backbone fusing image features into triplane latents | Image token sequence | triplane [3, C, H, W] |
52
+ | **synthesizer** | `synthesizer_*.gguf` | OSGDecoder: samples the triplane at voxel points and decodes | triplane + sample point coordinates | SDF / deformation / FlexiCubes weight / RGB |
53
+
54
+ **Model details**
55
+
56
+ - **dino**: splits the input image into patches and encodes them with self-attention
57
+ (flash attention), outputting a token sequence. It determines "how much detail the
58
+ model can see in the image" and is the feature source for all subsequent reconstruction.
59
+ - **lrm_transformer**: 16 layers of 1024-dim BasicTransformerBlock (self/cross-attention +
60
+ GELU MLP + adaLN modulation) projecting DINO features into three orthogonal feature
61
+ planes (triplane), implicitly encoding the full object in a compact low-dimensional space.
62
+ - **synthesizer**: given sample points, projects them onto the triplanes, concatenates
63
+ the features and decodes via a small MLP the point's **SDF** (signed distance),
64
+ **deformation** (vertex deformation), **weight** (FlexiCubes weights) and **RGB**
65
+ (`net_rgb` branch, shared by vertex colors and texture baking).
66
+
67
+ The three are chained: without dino there are no image features, without
68
+ lrm_transformer no triplane, without synthesizer no SDF/mesh/colors.
69
 
70
  ---
71
 
72
+ ## 3. File Size and Memory Footprint
73
 
74
+ | Model | f32 (MB) | f16 (MB) | q8 (MB) | f16/f32 | q8/f32 |
75
  |---|---|---|---|---|---|
76
  | dino | 459.0 | 230.1 | 123.4 | **0.50x** | **0.27x** |
77
  | lrm_transformer | 1054.8 | 527.8 | 284.0 | **0.50x** | **0.27x** |
78
  | synthesizer | 0.8 | 0.4 | 0.3 | **0.50x** | **0.32x** |
79
+ | **All three models** | **1514.6** | **758.3** | **407.7** | 0.50x | 0.27x |
80
 
81
+ - f16 is always half of f32 and q8 about a quarter, matching the theoretical
82
+ quantization size ratios.
83
+ - Size is **dominated by lrm_transformer** (~1 GB at f32): it drives most of the
84
+ end-to-end memory/load time. Measured ggml CUDA f16 end-to-end peak memory is
85
+ **1011 MB** (PyTorch under the same conditions: 3725 MB); an RTX 3060 with 12 GB
86
+ VRAM runs comfortably at grid_res=88.
87
 
88
  ---
89
 
90
+ ## 4. Performance (end-to-end, seconds per image)
91
 
92
+ | Backend | f32 | f16 | q8 | Notes |
93
  |---|---|---|---|---|
94
+ | **CUDA** (RTX 3060) | 3.5 | **3.0** | **2.9** | Recommended; **4.5x** faster than PyTorch CUDA (13.5s f16) |
95
+ | **Vulkan** | 4.0 | 3.5 | 3.4 | Preferred path for cross-vendor GPUs (AMD/Intel) |
96
+ | **CPU** (Ryzen 5950X) | 24.8 | 24.4 | 23.5 | Usable without a GPU; **avoid f16** |
97
+ | PyTorch CUDA (official) | 13.7 | 13.5 | — | Reference baseline |
98
+
99
+ - On GPUs, **q8 is the fastest**, f16 next; CUDA is ~10–15% faster than Vulkan.
100
+ - **On CPU, f16 is actually the slowest/unstable**: caused by the CPU-side f16→f32
101
+ dequantization overhead; choose f32 or q8 on CPU.
102
+ - Per-stage timing breakdown (ggml CUDA f16): see
103
+ `../benchmarks/results/perf_ggml_stage_breakdown.png`; lrm_transformer dominates,
104
+ dino is next, and synthesizer takes a tiny share.
105
 
106
  ---
107
 
108
+ ## 5. Accuracy (SDF relative error, mean|sdf|≈8.5, grid_res=88)
109
 
110
+ ### 5.1 Pure quantization error (same backend, vs each backend's own f32)
111
 
112
+ | Backend | f16 relative RMSE | q8 relative RMSE |
113
  |---|---|---|
114
  | CPU | 0.18% | 2.86% |
115
  | CUDA | 0.81% | 2.83% |
116
  | Vulkan | 0.76% | 2.00% |
117
 
118
+ ### 5.2 Backend consistency (f32 vs CUDA-f32)
119
 
120
+ | Backend | Relative RMSE |
121
  |---|---|
122
  | CPU | 0.64% |
123
  | Vulkan | 0.72% |
124
 
125
+ - **f16 is nearly lossless** (relative error <1%), mesh geometry unchanged; **q8 is
126
+ acceptable** (2–3%), SDF sign-flip rate <0.8%, geometry essentially indistinguishable
127
+ to the eye.
128
+ - **Across the three backends, same-precision results agree within <1%**: quantization
129
+ and backend implementations are consistent — no "backend silently losing accuracy".
130
 
131
  ---
132
 
133
+ ## 6. Recommended Use Cases
134
 
135
+ | Scenario | Recommendation | Rationale |
136
  |---|---|---|
137
+ | Everyday inference on NVIDIA GPUs | **CUDA + f16** | <1% accuracy loss, 3.0 s/image, ~1 GB peak memory |
138
+ | Maximum speed / batch throughput | CUDA + **q8** | Fastest (2.9 s), smallest size (408 MB for the trio), 2–3% accuracy acceptable |
139
+ | AMD / Intel / GPUs without CUDA drivers | **Vulkan + f16** | The only cross-vendor GPU path, 3.5 s/image |
140
+ | Tight VRAM (≤4 GB) or mobile | **q8** (lrm_transformer only 284 MB) | Smallest peak memory |
141
+ | Server CPU deployment | CPU + **f32 or q8** | Avoids the slow CPU f16 dequantization pitfall |
142
+ | Accuracy benchmarking / regression tests / research reproduction | **f32** | Layer-by-layer comparable with the PyTorch reference; serves as the parity reference |
143
+ | Validating the pipeline on edge devices | **q8 CPU** | No GPU dependency at all, runs at ~24 s/image |
144
+
145
+ **Per-model advice**
146
+
147
+ - **dino**: determines feature quality; generally **not lower than f16**. Use q8 only
148
+ for maximum speed when a slight quality loss is acceptable. Input resolution is
149
+ fixed at 224×224.
150
+ - **lrm_transformer**: the largest model, where quantization gains and risks are most
151
+ concentrated; if q8 shows geometric distortion, fall back to f16.
152
+ - **synthesizer**: <1 MB, quantization gains are negligible; **prefer f16 or f32
153
+ directly** to avoid pointless accuracy loss; its `plane_dim` must match the
154
+ lrm_transformer output.
155
 
156
  ---
157
 
158
+ ## 7. Reproduction and Extension
159
+
160
+ - Batch runner: `bash ../benchmarks/run_bench.sh` (3 backends × 3 precisions × 4 demos,
161
+ producing `times.csv` and per-run `.sdf.bin` / `.obj`)
162
+ - Analysis/plots: `python3 ../benchmarks/analyze.py` (generates `report.md`,
163
+ `latency.png`, `precision_rmse.png`, `precision_maxerr.png`)
164
+ - Weight conversion (official ckpt → GGUF, three precisions): `python3 -m convert.convert_all`
165
+ - Manual inference: see the full commands in [README.md §3 Step 5](../README.md) at the
166
+ module root.
167