InstantMesh-GUFF / README.md
Asher-1's picture
Create README.md
713ccfb verified
|
Raw History Blame Contribute Delete
7.69 kB
# MODEL_CARD β€” InstantMesh GGUF Inference Model Card
This directory (`cpp_ggml/models/gguf/`) holds the 3 ggml models required by the
InstantMesh end-to-end reconstruction pipeline, each provided as `.gguf` weights in
three precisions: `f32 / f16 / q8`.
**Weight download (Hugging Face): https://huggingface.co/Asher-1/InstantMeshGGuf**
Pipeline and data flow:
```
Image β†’ β‘  dino (image encoding) β†’ β‘‘ lrm_transformer (triplane generation)
β†’ β‘’ synthesizer (OSGDecoder voxel decoding: SDF/deform/weight/RGB)
β†’ β‘£ flexicubes (SDF β†’ mesh, no weights)
```
> Performance and accuracy data in this card come from `../benchmarks/results/report.md`
> (RTX 3060 / Ryzen 5950X, grid_res=88, same conditions as the PyTorch reference;
> end-to-end includes model loading and mesh export).
---
## 1. Download and Usage
```bash
# All 9 files (~2.7 GB)
huggingface-cli download Asher-1/InstantMeshGGuf --local-dir cpp_ggml/models/gguf
# Or the minimal usable set: the f16 trio (~760 MB)
for f in dino_f16 lrm_transformer_f16 synthesizer_f16; do
curl -L -o cpp_ggml/models/gguf/$f.gguf \
https://huggingface.co/Asher-1/InstantMeshGGuf/resolve/main/$f.gguf
done
```
- The three models **must use the same precision** (`dino_f16 + lrm_transformer_f16 +
synthesizer_f16`); mixing precisions may cause inter-layer numeric mismatches.
- `.gguf` files are not committed (large binaries); the weights contain network
parameters only, **no preprocessing parameters**. Inputs must be ImageNet-normalized
per DINOv2 rules (see `../src/tools/instantmesh.cpp` and `../convert/prep_input.py`).
---
## 2. Model Overview
| Model | File prefix | Role | Input | Output |
|---|---|---|---|---|
| **dino** | `dino_*.gguf` | DINOv2 vision Transformer backbone extracting multi-scale patch features | Multi-view RGB tensor [V,3,224,224] (ImageNet-normalized) | Image token sequence |
| **lrm_transformer** | `lrm_transformer_*.gguf` | Reconstruction backbone fusing image features into triplane latents | Image token sequence | triplane [3, C, H, W] |
| **synthesizer** | `synthesizer_*.gguf` | OSGDecoder: samples the triplane at voxel points and decodes | triplane + sample point coordinates | SDF / deformation / FlexiCubes weight / RGB |
**Model details**
- **dino**: splits the input image into patches and encodes them with self-attention
(flash attention), outputting a token sequence. It determines "how much detail the
model can see in the image" and is the feature source for all subsequent reconstruction.
- **lrm_transformer**: 16 layers of 1024-dim BasicTransformerBlock (self/cross-attention +
GELU MLP + adaLN modulation) projecting DINO features into three orthogonal feature
planes (triplane), implicitly encoding the full object in a compact low-dimensional space.
- **synthesizer**: given sample points, projects them onto the triplanes, concatenates
the features and decodes via a small MLP the point's **SDF** (signed distance),
**deformation** (vertex deformation), **weight** (FlexiCubes weights) and **RGB**
(`net_rgb` branch, shared by vertex colors and texture baking).
The three are chained: without dino there are no image features, without
lrm_transformer no triplane, without synthesizer no SDF/mesh/colors.
---
## 3. File Size and Memory Footprint
| Model | f32 (MB) | f16 (MB) | q8 (MB) | f16/f32 | q8/f32 |
|---|---|---|---|---|---|
| dino | 459.0 | 230.1 | 123.4 | **0.50x** | **0.27x** |
| lrm_transformer | 1054.8 | 527.8 | 284.0 | **0.50x** | **0.27x** |
| synthesizer | 0.8 | 0.4 | 0.3 | **0.50x** | **0.32x** |
| **All three models** | **1514.6** | **758.3** | **407.7** | 0.50x | 0.27x |
- f16 is always half of f32 and q8 about a quarter, matching the theoretical
quantization size ratios.
- Size is **dominated by lrm_transformer** (~1 GB at f32): it drives most of the
end-to-end memory/load time. Measured ggml CUDA f16 end-to-end peak memory is
**1011 MB** (PyTorch under the same conditions: 3725 MB); an RTX 3060 with 12 GB
VRAM runs comfortably at grid_res=88.
---
## 4. Performance (end-to-end, seconds per image)
| Backend | f32 | f16 | q8 | Notes |
|---|---|---|---|---|
| **CUDA** (RTX 3060) | 3.5 | **3.0** | **2.9** | Recommended; **4.5x** faster than PyTorch CUDA (13.5s f16) |
| **Vulkan** | 4.0 | 3.5 | 3.4 | Preferred path for cross-vendor GPUs (AMD/Intel) |
| **CPU** (Ryzen 5950X) | 24.8 | 24.4 | 23.5 | Usable without a GPU; **avoid f16** |
| PyTorch CUDA (official) | 13.7 | 13.5 | β€” | Reference baseline |
- On GPUs, **q8 is the fastest**, f16 next; CUDA is ~10–15% faster than Vulkan.
- **On CPU, f16 is actually the slowest/unstable**: caused by the CPU-side f16β†’f32
dequantization overhead; choose f32 or q8 on CPU.
- Per-stage timing breakdown (ggml CUDA f16): see
`../benchmarks/results/perf_ggml_stage_breakdown.png`; lrm_transformer dominates,
dino is next, and synthesizer takes a tiny share.
---
## 5. Accuracy (SDF relative error, mean|sdf|β‰ˆ8.5, grid_res=88)
### 5.1 Pure quantization error (same backend, vs each backend's own f32)
| Backend | f16 relative RMSE | q8 relative RMSE |
|---|---|---|
| CPU | 0.18% | 2.86% |
| CUDA | 0.81% | 2.83% |
| Vulkan | 0.76% | 2.00% |
### 5.2 Backend consistency (f32 vs CUDA-f32)
| Backend | Relative RMSE |
|---|---|
| CPU | 0.64% |
| Vulkan | 0.72% |
- **f16 is nearly lossless** (relative error <1%), mesh geometry unchanged; **q8 is
acceptable** (2–3%), SDF sign-flip rate <0.8%, geometry essentially indistinguishable
to the eye.
- **Across the three backends, same-precision results agree within <1%**: quantization
and backend implementations are consistent β€” no "backend silently losing accuracy".
---
## 6. Recommended Use Cases
| Scenario | Recommendation | Rationale |
|---|---|---|
| Everyday inference on NVIDIA GPUs | **CUDA + f16** | <1% accuracy loss, 3.0 s/image, ~1 GB peak memory |
| Maximum speed / batch throughput | CUDA + **q8** | Fastest (2.9 s), smallest size (408 MB for the trio), 2–3% accuracy acceptable |
| AMD / Intel / GPUs without CUDA drivers | **Vulkan + f16** | The only cross-vendor GPU path, 3.5 s/image |
| Tight VRAM (≀4 GB) or mobile | **q8** (lrm_transformer only 284 MB) | Smallest peak memory |
| Server CPU deployment | CPU + **f32 or q8** | Avoids the slow CPU f16 dequantization pitfall |
| Accuracy benchmarking / regression tests / research reproduction | **f32** | Layer-by-layer comparable with the PyTorch reference; serves as the parity reference |
| Validating the pipeline on edge devices | **q8 CPU** | No GPU dependency at all, runs at ~24 s/image |
**Per-model advice**
- **dino**: determines feature quality; generally **not lower than f16**. Use q8 only
for maximum speed when a slight quality loss is acceptable. Input resolution is
fixed at 224Γ—224.
- **lrm_transformer**: the largest model, where quantization gains and risks are most
concentrated; if q8 shows geometric distortion, fall back to f16.
- **synthesizer**: <1 MB, quantization gains are negligible; **prefer f16 or f32
directly** to avoid pointless accuracy loss; its `plane_dim` must match the
lrm_transformer output.
---
## 7. Reproduction and Extension
- Batch runner: `bash ../benchmarks/run_bench.sh` (3 backends Γ— 3 precisions Γ— 4 demos,
producing `times.csv` and per-run `.sdf.bin` / `.obj`)
- Analysis/plots: `python3 ../benchmarks/analyze.py` (generates `report.md`,
`latency.png`, `precision_rmse.png`, `precision_maxerr.png`)
- Weight conversion (official ckpt β†’ GGUF, three precisions): `python3 -m convert.convert_all`
- Manual inference: see the full commands in [README.md Β§3 Step 5](../README.md) at the
module root.