Flor commited on
Commit
1c8628c
·
verified ·
1 Parent(s): 0ccdc61

card: update from fastforks 1cbf1671

Browse files
Files changed (1) hide show
  1. README.md +105 -0
README.md ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: "fastforks"
3
+ pipeline_tag: "image-to-3d"
4
+ license: "mit"
5
+ tags:
6
+ - "multiview"
7
+ - "multi-view-stereo"
8
+ - "3d-reconstruction"
9
+ - "camera-pose-estimation"
10
+ - "cuda"
11
+ - "pytorch"
12
+ - "fastforks"
13
+ - "fastforks-card"
14
+ ---
15
+ # MVSplat (fastforks)
16
+
17
+ MVSplat (Chen et al. 2024): feed-forward 3D Gaussian splatting and depth from posed, calibrated images. A UniMatch CNN and multi-view swin transformer feed a plane-sweep cost volume (all other views), refined by 2D U-Nets with cross-view attention into every view's depth; a Gaussian head predicts one Gaussian per pixel (opacity, scale, rotation, degree-4 SH) in the given world frame (`result.gaussians`, `.save("scene.ply")`). Forward pass only: no renderer.
18
+
19
+ This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/donydchen/mvsplat> at `01f9a28`.
20
+
21
+ Task: multiview. Tier: experimental. Outputs: `depth`, `confidence`, `gaussians`. Backends: `portable`, `kernels`.
22
+
23
+ Input: several images of one scene (posed and calibrated for multi-view stereo, unposed otherwise). Output: depth maps and, where the model predicts them, cameras and 3-D points.
24
+
25
+ ## Variants
26
+
27
+ Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/multiview/models/mvsplat/card.toml).
28
+
29
+ | variant | weights (author link) | SHA-256 | weights licence | commercial use |
30
+ |---|---|---|---|---|
31
+ | `mvsplat-re10k@1` | [re10k.ckpt](https://drive.usercontent.google.com/download?id=1Y5OAJN9AX5yMjkbyxag5OnwacdjPnl-N&export=download&confirm=t) | `83d0d9eaa753` | MIT | yes |
32
+
33
+ ## Licence
34
+
35
+ Original-code strings, recorded as they are (research-use package; check them before any other use). Code: MIT. Licence files: `LICENSE`.
36
+
37
+ Training data: RealEstate10K (camera trajectories of YouTube real-estate videos) (RealEstate10K (YouTube videos, camera poses from ORB-SLAM; research use per its release), the backbone initialised from UniMatch (GMDepth, ScanNet).).
38
+
39
+ - Licence note: Only the RealEstate10K checkpoint is shipped (MIT code and weights); the ACID checkpoint has the same architecture.
40
+
41
+ Licence evidence: https://github.com/donydchen/mvsplat/blob/01f9a28edb5eb68416e7e63b01f8d90c3bdfbf01/LICENSE and https://github.com/donydchen/mvsplat/blob/01f9a28edb5eb68416e7e63b01f8d90c3bdfbf01/README.md and https://drive.google.com/drive/folders/14_E_5R6ojOWnLSrSVLVEMHnTiKsfddjU
42
+
43
+ ## Input size
44
+
45
+ any H x W >= 1, resized or padded inside the model
46
+ Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `mvsplat-re10k` 3000x4000 (3.2 GiB peak allocated, 4.4 reserved)
47
+
48
+ ## Speed and memory
49
+
50
+ ### Multiview, synthetic
51
+
52
+ | model | input | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
53
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
54
+ | `mvsplat-re10k` | 336x518 | – | 279 | 181 | 172 | 1.6x | 1.0x | 2889 / 4160 | 1239 / 1532 | 1239 / 1532 |
55
+
56
+ ### Startup and first call
57
+
58
+ | model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
59
+ | --- | --- | --- | --- | --- | --- | --- |
60
+ | `mvsplat-re10k` | 1.37 (336x518) | 0.15 (336x518) | 1.27 (336x518) | 2.82 (336x518) | 1.02 (336x518) | 0.69 (336x518) |
61
+
62
+ ### torch.compile and TensorRT
63
+
64
+ | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
65
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
66
+ | `mvsplat-re10k` | 336x518 | 181 ms | 115 ms | planned for 0.2.1 | 172 ms | 102 s | – | 1239 / 1532 | 2762 / 2810 | 1239 / 1532 | – |
67
+
68
+ Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
69
+
70
+ Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
71
+
72
+ ## Accuracy
73
+
74
+ ### On public datasets
75
+
76
+ 640x512 crops, the 7 most co-visible views as sources; GT = depth rendered from the structured-light points. AbsRel / delta1 over GT pixels, after per-image median scaling unless the model takes the cameras (calibrated MVS: cameras, error in mm; PatchMatchNet also the depth range 425-935 mm, MVSAnywhere its own range from the camera baselines). Original = the original code named under Details, same checkpoint and protocol.
77
+
78
+ | variant | dataset | original fp32 | original fp16 | portable | kernels | note |
79
+ |---|---|---|---|---|---|---|
80
+ | `mvsplat-re10k` | DTU scans 1, 9, 24, 33, 75, view 24 + 7 sources, AbsRel / delta1 % / MAE mm | 0.0438 / 97.31 / 31.17 | 0.0439 / 97.30 / 31.21 | 0.0438 / 97.30 / 31.20 | 0.0438 / 97.30 / 31.20 | |
81
+
82
+ Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 28db82cb, 2026-11-13.
83
+
84
+ ### Parity with the original code
85
+
86
+ Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.02 and p99 <= 0.25 (absolute error of the output `depth`).
87
+
88
+ ## Usage
89
+
90
+ ```python
91
+ from fastforks import multiview
92
+
93
+ model = multiview.load("mvsplat-re10k") # mode="auto": the preferred backend from recorded measurements
94
+ result = model(["view0.jpg", "view1.jpg", "view2.jpg"]) # the views of one scene, no cameras needed
95
+ result.depth # [V, H, W] float32 tensor; result.kind says what the maps mean
96
+ result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
97
+ ```
98
+
99
+ Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `mvsplat-re10k`.
100
+
101
+ ## Kernels backend
102
+
103
+ `multiview.load("mvsplat-re10k", mode="kernels")` runs our CUDA kernels (`cudnn_hacc`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
104
+
105
+ <!-- staged from fastforks 1cbf1671 (2026-11-15) -->