# MODEL_CARD — InstantMesh GGUF Inference Model Card This directory (`cpp_ggml/models/gguf/`) holds the 3 ggml models required by the InstantMesh end-to-end reconstruction pipeline, each provided as `.gguf` weights in three precisions: `f32 / f16 / q8`. **Weight download (Hugging Face): https://huggingface.co/Asher-1/InstantMeshGGuf** Pipeline and data flow: ``` Image → ① dino (image encoding) → ② lrm_transformer (triplane generation) → ③ synthesizer (OSGDecoder voxel decoding: SDF/deform/weight/RGB) → ④ flexicubes (SDF → mesh, no weights) ``` > Performance and accuracy data in this card come from `../benchmarks/results/report.md` > (RTX 3060 / Ryzen 5950X, grid_res=88, same conditions as the PyTorch reference; > end-to-end includes model loading and mesh export). --- ## 1. Download and Usage ```bash # All 9 files (~2.7 GB) huggingface-cli download Asher-1/InstantMeshGGuf --local-dir cpp_ggml/models/gguf # Or the minimal usable set: the f16 trio (~760 MB) for f in dino_f16 lrm_transformer_f16 synthesizer_f16; do curl -L -o cpp_ggml/models/gguf/$f.gguf \ https://huggingface.co/Asher-1/InstantMeshGGuf/resolve/main/$f.gguf done ``` - The three models **must use the same precision** (`dino_f16 + lrm_transformer_f16 + synthesizer_f16`); mixing precisions may cause inter-layer numeric mismatches. - `.gguf` files are not committed (large binaries); the weights contain network parameters only, **no preprocessing parameters**. Inputs must be ImageNet-normalized per DINOv2 rules (see `../src/tools/instantmesh.cpp` and `../convert/prep_input.py`). --- ## 2. Model Overview | Model | File prefix | Role | Input | Output | |---|---|---|---|---| | **dino** | `dino_*.gguf` | DINOv2 vision Transformer backbone extracting multi-scale patch features | Multi-view RGB tensor [V,3,224,224] (ImageNet-normalized) | Image token sequence | | **lrm_transformer** | `lrm_transformer_*.gguf` | Reconstruction backbone fusing image features into triplane latents | Image token sequence | triplane [3, C, H, W] | | **synthesizer** | `synthesizer_*.gguf` | OSGDecoder: samples the triplane at voxel points and decodes | triplane + sample point coordinates | SDF / deformation / FlexiCubes weight / RGB | **Model details** - **dino**: splits the input image into patches and encodes them with self-attention (flash attention), outputting a token sequence. It determines "how much detail the model can see in the image" and is the feature source for all subsequent reconstruction. - **lrm_transformer**: 16 layers of 1024-dim BasicTransformerBlock (self/cross-attention + GELU MLP + adaLN modulation) projecting DINO features into three orthogonal feature planes (triplane), implicitly encoding the full object in a compact low-dimensional space. - **synthesizer**: given sample points, projects them onto the triplanes, concatenates the features and decodes via a small MLP the point's **SDF** (signed distance), **deformation** (vertex deformation), **weight** (FlexiCubes weights) and **RGB** (`net_rgb` branch, shared by vertex colors and texture baking). The three are chained: without dino there are no image features, without lrm_transformer no triplane, without synthesizer no SDF/mesh/colors. --- ## 3. File Size and Memory Footprint | Model | f32 (MB) | f16 (MB) | q8 (MB) | f16/f32 | q8/f32 | |---|---|---|---|---|---| | dino | 459.0 | 230.1 | 123.4 | **0.50x** | **0.27x** | | lrm_transformer | 1054.8 | 527.8 | 284.0 | **0.50x** | **0.27x** | | synthesizer | 0.8 | 0.4 | 0.3 | **0.50x** | **0.32x** | | **All three models** | **1514.6** | **758.3** | **407.7** | 0.50x | 0.27x | - f16 is always half of f32 and q8 about a quarter, matching the theoretical quantization size ratios. - Size is **dominated by lrm_transformer** (~1 GB at f32): it drives most of the end-to-end memory/load time. Measured ggml CUDA f16 end-to-end peak memory is **1011 MB** (PyTorch under the same conditions: 3725 MB); an RTX 3060 with 12 GB VRAM runs comfortably at grid_res=88. --- ## 4. Performance (end-to-end, seconds per image) | Backend | f32 | f16 | q8 | Notes | |---|---|---|---|---| | **CUDA** (RTX 3060) | 3.5 | **3.0** | **2.9** | Recommended; **4.5x** faster than PyTorch CUDA (13.5s f16) | | **Vulkan** | 4.0 | 3.5 | 3.4 | Preferred path for cross-vendor GPUs (AMD/Intel) | | **CPU** (Ryzen 5950X) | 24.8 | 24.4 | 23.5 | Usable without a GPU; **avoid f16** | | PyTorch CUDA (official) | 13.7 | 13.5 | — | Reference baseline | - On GPUs, **q8 is the fastest**, f16 next; CUDA is ~10–15% faster than Vulkan. - **On CPU, f16 is actually the slowest/unstable**: caused by the CPU-side f16→f32 dequantization overhead; choose f32 or q8 on CPU. - Per-stage timing breakdown (ggml CUDA f16): see `../benchmarks/results/perf_ggml_stage_breakdown.png`; lrm_transformer dominates, dino is next, and synthesizer takes a tiny share. --- ## 5. Accuracy (SDF relative error, mean|sdf|≈8.5, grid_res=88) ### 5.1 Pure quantization error (same backend, vs each backend's own f32) | Backend | f16 relative RMSE | q8 relative RMSE | |---|---|---| | CPU | 0.18% | 2.86% | | CUDA | 0.81% | 2.83% | | Vulkan | 0.76% | 2.00% | ### 5.2 Backend consistency (f32 vs CUDA-f32) | Backend | Relative RMSE | |---|---| | CPU | 0.64% | | Vulkan | 0.72% | - **f16 is nearly lossless** (relative error <1%), mesh geometry unchanged; **q8 is acceptable** (2–3%), SDF sign-flip rate <0.8%, geometry essentially indistinguishable to the eye. - **Across the three backends, same-precision results agree within <1%**: quantization and backend implementations are consistent — no "backend silently losing accuracy". --- ## 6. Recommended Use Cases | Scenario | Recommendation | Rationale | |---|---|---| | Everyday inference on NVIDIA GPUs | **CUDA + f16** | <1% accuracy loss, 3.0 s/image, ~1 GB peak memory | | Maximum speed / batch throughput | CUDA + **q8** | Fastest (2.9 s), smallest size (408 MB for the trio), 2–3% accuracy acceptable | | AMD / Intel / GPUs without CUDA drivers | **Vulkan + f16** | The only cross-vendor GPU path, 3.5 s/image | | Tight VRAM (≤4 GB) or mobile | **q8** (lrm_transformer only 284 MB) | Smallest peak memory | | Server CPU deployment | CPU + **f32 or q8** | Avoids the slow CPU f16 dequantization pitfall | | Accuracy benchmarking / regression tests / research reproduction | **f32** | Layer-by-layer comparable with the PyTorch reference; serves as the parity reference | | Validating the pipeline on edge devices | **q8 CPU** | No GPU dependency at all, runs at ~24 s/image | **Per-model advice** - **dino**: determines feature quality; generally **not lower than f16**. Use q8 only for maximum speed when a slight quality loss is acceptable. Input resolution is fixed at 224×224. - **lrm_transformer**: the largest model, where quantization gains and risks are most concentrated; if q8 shows geometric distortion, fall back to f16. - **synthesizer**: <1 MB, quantization gains are negligible; **prefer f16 or f32 directly** to avoid pointless accuracy loss; its `plane_dim` must match the lrm_transformer output. --- ## 7. Reproduction and Extension - Batch runner: `bash ../benchmarks/run_bench.sh` (3 backends × 3 precisions × 4 demos, producing `times.csv` and per-run `.sdf.bin` / `.obj`) - Analysis/plots: `python3 ../benchmarks/analyze.py` (generates `report.md`, `latency.png`, `precision_rmse.png`, `precision_maxerr.png`) - Weight conversion (official ckpt → GGUF, three precisions): `python3 -m convert.convert_all` - Manual inference: see the full commands in [README.md §3 Step 5](../README.md) at the module root.