Flor commited on
Commit
d9cf2e0
Β·
verified Β·
1 Parent(s): fa77335

card: update from fastforks 1cbf1671

Browse files
Files changed (1) hide show
  1. README.md +152 -0
README.md ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: "fastforks"
3
+ pipeline_tag: "image-to-image"
4
+ license: "other"
5
+ license_name: "see-model-card"
6
+ license_link: "https://github.com/autonomousvision/unimatch/blob/943a99006950ab6c84ea72a72beb147fdd03c19a/MODEL_ZOO.md"
7
+ tags:
8
+ - "flow"
9
+ - "optical-flow"
10
+ - "motion-estimation"
11
+ - "cuda"
12
+ - "pytorch"
13
+ - "fastforks"
14
+ - "fastforks-card"
15
+ ---
16
+ # GMFlow (fastforks)
17
+
18
+ GMFlow (Xu, Zhang, Cai, Rezatofighi & Tao 2022) in its UniMatch release (Xu et al. 2023): optical flow as global matching. A CNN gives features at 1/8 resolution, a Transformer with self- and cross-attention (shifted windows) makes the two images' features aware of each other, the flow is the expected position of the best match over the whole image, and a self-attention layer spreads it into occluded areas. `scale2` variants repeat this at 1/4 resolution with local matching; `regrefine6` variants add 6 RAFT-like local refinements.
19
+
20
+ This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/autonomousvision/unimatch> at `943a99006950`.
21
+
22
+ Task: flow. Tier: experimental. Outputs: `flow`. Backends: `portable`.
23
+
24
+ Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).
25
+
26
+ ## Variants
27
+
28
+ Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/flow/models/gmflow/card.toml).
29
+
30
+ | variant | weights (author link) | SHA-256 | weights licence | commercial use |
31
+ |---|---|---|---|---|
32
+ | `gmflow-scale1-mixdata@1` | [gmflow-scale1-mixdata-train320x576-4c3a6e9a.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale1-mixdata-train320x576-4c3a6e9a.pth) | `4c3a6e9af913` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
33
+ | `gmflow-scale1-things@1` | [gmflow-scale1-things-e9887eda.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale1-things-e9887eda.pth) | `e9887eda6165` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
34
+ | `gmflow-scale2-mixdata@1` | [gmflow-scale2-mixdata-train320x576-9ff1c094.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-mixdata-train320x576-9ff1c094.pth) | `9ff1c094f496` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
35
+ | `gmflow-scale2-regrefine6-kitti@1` | [gmflow-scale2-regrefine6-kitti15-25b554d7.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-kitti15-25b554d7.pth) | `25b554d766e9` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
36
+ | `gmflow-scale2-regrefine6-mixdata@1` | [gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth) | `4e7b215d5a25` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
37
+ | `gmflow-scale2-regrefine6-sintelft@1` | [gmflow-scale2-regrefine6-sintelft-6e39e2b9.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-sintelft-6e39e2b9.pth) | `6e39e2b9fb0f` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
38
+ | `gmflow-scale2-regrefine6-things@1` | [gmflow-scale2-regrefine6-things-776ed612.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-things-776ed612.pth) | `776ed612efb8` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
39
+ | `gmflow-scale2-sintel@1` | [gmflow-scale2-sintel-3ed1cf48.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-sintel-3ed1cf48.pth) | `3ed1cf486cf7` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
40
+ | `gmflow-scale2-things@1` | [gmflow-scale2-things-36579974.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-things-36579974.pth) | `365799747d9f` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
41
+
42
+ ## Licence
43
+
44
+ Original-code strings, recorded as they are (research-use package; check them before any other use). Code: MIT AND Apache-2.0 AND BSD-3-Clause. Licence files: `LICENSE`, `LICENSE-Apache-2.0`, `LICENSE-RAFT`.
45
+
46
+ Code licence: MIT AND Apache-2.0 AND BSD-3-Clause; the authors do not state a separate weights licence.
47
+
48
+ Training data: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa) (FlyingChairs, then FlyingThings3D (part of SceneFlow)).
49
+
50
+ Licence evidence: <https://github.com/autonomousvision/unimatch/blob/943a99006950ab6c84ea72a72beb147fdd03c19a/MODEL_ZOO.md>
51
+
52
+ ## Input size
53
+
54
+ - `gmflow-scale1-mixdata`, `gmflow-scale1-things`: any H x W >= 64, padded to a multiple of 16
55
+ - `gmflow-scale2-mixdata`, `gmflow-scale2-regrefine6-kitti`, `gmflow-scale2-regrefine6-mixdata`, `gmflow-scale2-regrefine6-sintelft`, `gmflow-scale2-regrefine6-things`, `gmflow-scale2-sintel`, `gmflow-scale2-things`: any H x W >= 64, padded to a multiple of 32
56
+ Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `gmflow-scale1-mixdata` 1024x2048 (5.4 GiB peak allocated, 6.0 reserved); `gmflow-scale1-things` 1024x2048 (5.4 GiB peak allocated, 6.0 reserved); `gmflow-scale2-mixdata` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-kitti` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-mixdata` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-sintelft` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-things` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-sintel` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-things` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved)
57
+
58
+ ## Speed and memory
59
+
60
+ ### Flow, synthetic
61
+
62
+ | model | input | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
63
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
64
+ | `gmflow-scale1-mixdata` | 1080x1920 | 1246 | 750 | 556 | – | – | – | 8360 / 8448 | 4496 / 4642 | – |
65
+ | `gmflow-scale1-things` | 1080x1920 | 1245 | 750 | 556 | – | – | – | 8360 / 8468 | 4496 / 4642 | – |
66
+ | `gmflow-scale2-mixdata` | 1080x1920 | 2544 | 1608 | 1139 | – | – | – | 8425 / 8572 | 6017 / 6224 | – |
67
+ | `gmflow-scale2-regrefine6-kitti` | 1080x1920 | 3255 | 2188 | 1678 | – | – | – | 8537 / 8822 | 6080 / 6230 | – |
68
+ | `gmflow-scale2-regrefine6-mixdata` | 1080x1920 | 3254 | 2188 | 1675 | – | – | – | 8537 / 8822 | 6080 / 6250 | – |
69
+ | `gmflow-scale2-regrefine6-sintelft` | 1080x1920 | 3255 | 2184 | 1674 | – | – | – | 8537 / 8822 | 6080 / 6250 | – |
70
+ | `gmflow-scale2-regrefine6-things` | 1080x1920 | 3230 | 2175 | 1665 | – | – | – | 8537 / 8822 | 6080 / 6230 | – |
71
+ | `gmflow-scale2-sintel` | 1080x1920 | 2541 | 1607 | 1139 | – | – | – | 8425 / 8552 | 6017 / 6224 | – |
72
+ | `gmflow-scale2-things` | 1080x1920 | 2537 | 1599 | 1139 | – | – | – | 8425 / 8552 | 6017 / 6224 | – |
73
+
74
+ ### Startup and first call
75
+
76
+ | model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
77
+ | --- | --- | --- | --- | --- | --- | --- |
78
+ | `gmflow-scale1-mixdata` | 0.13 | 0.16 | – | 0.76 | 0.56 | – |
79
+ | `gmflow-scale1-things` | 0.14 | 0.16 | – | 0.94 | 0.56 | – |
80
+ | `gmflow-scale2-mixdata` | 0.13 | 0.16 | – | 1.62 | 1.14 | – |
81
+ | `gmflow-scale2-regrefine6-kitti` | 0.15 | 0.46 | – | 2.20 | 1.68 | – |
82
+ | `gmflow-scale2-regrefine6-mixdata` | 0.14 | 0.25 | – | 2.20 | 1.68 | – |
83
+ | `gmflow-scale2-regrefine6-sintelft` | 0.15 | 0.26 | – | 2.37 | 1.68 | – |
84
+ | `gmflow-scale2-regrefine6-things` | 0.15 | 0.16 | – | 2.19 | 1.67 | – |
85
+ | `gmflow-scale2-things` | 0.14 | 0.17 | – | 1.61 | 1.14 | – |
86
+
87
+ ### torch.compile and TensorRT
88
+
89
+ | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
90
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
91
+ | `gmflow-scale1-mixdata` | 1080x1920 | 556 ms | 382 ms | 187 ms | – | 29.1 s | 67.3 s | 4496 / 4642 | 4293 / 4558 | – | 4333 / 4390 |
92
+ | `gmflow-scale1-things` | 1080x1920 | 556 ms | 382 ms | 187 ms | – | 35.3 s | 69.3 s | 4496 / 4642 | 4293 / 4558 | – | 4333 / 4390 |
93
+ | `gmflow-scale2-mixdata` | 1080x1920 | 1139 ms | 765 ms | planned for 0.2.1 | – | 46.6 s | – | 6017 / 6224 | 10865 / 10904 | – | – |
94
+ | `gmflow-scale2-regrefine6-kitti` | 1080x1920 | 1678 ms | 1116 ms | planned for 0.2.1 | – | 58.4 s | – | 6080 / 6230 | 10875 / 10972 | – | – |
95
+ | `gmflow-scale2-regrefine6-mixdata` | 1080x1920 | 1675 ms | 1116 ms | planned for 0.2.1 | – | 58.9 s | – | 6080 / 6250 | 10875 / 10972 | – | – |
96
+ | `gmflow-scale2-regrefine6-sintelft` | 1080x1920 | 1674 ms | 1116 ms | planned for 0.2.1 | – | 58.5 s | – | 6080 / 6250 | 10875 / 10972 | – | – |
97
+ | `gmflow-scale2-regrefine6-things` | 1080x1920 | 1665 ms | 1116 ms | planned for 0.2.1 | – | 58.2 s | – | 6080 / 6230 | 10875 / 10972 | – | – |
98
+ | `gmflow-scale2-things` | 1080x1920 | 1139 ms | 765 ms | planned for 0.2.1 | – | 46.7 s | – | 6017 / 6224 | 10865 / 10904 | – | – |
99
+
100
+ Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
101
+
102
+ - `gmflow-scale2-sintel`: same network and settings as `gmflow-scale2-mixdata`, measured once (rows not repeated).
103
+
104
+ Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
105
+
106
+ ## Accuracy
107
+
108
+ ### On public datasets
109
+
110
+ Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.
111
+
112
+ | variant | dataset | original fp32 | original fp16 | portable | note |
113
+ |---|---|---|---|---|---|
114
+ | `gmflow-scale1-things` | KITTI 2015 train, EPE / Fl-all % | 10.347 / 34.31 | 10.347 / 34.30 | 10.346 / 34.30 | |
115
+ | `gmflow-scale1-mixdata` | KITTI 2015 train, EPE / Fl-all % | 3.698 / 17.60 | 3.698 / 17.61 | 3.697 / 17.61 | possibly in the training data |
116
+ | `gmflow-scale2-things` | KITTI 2015 train, EPE / Fl-all % | 6.747 / 22.09 | 6.747 / 22.09 | 6.747 / 22.09 | |
117
+ | `gmflow-scale2-sintel` | KITTI 2015 train, EPE / Fl-all % | 1.975 / 7.88 | 1.976 / 7.88 | 1.975 / 7.88 | trained on this dataset's public images; not a zero-shot result |
118
+ | `gmflow-scale2-mixdata` | KITTI 2015 train, EPE / Fl-all % | 2.390 / 10.10 | 2.390 / 10.10 | 2.389 / 10.10 | possibly in the training data |
119
+ | `gmflow-scale2-regrefine6-things` | KITTI 2015 train, EPE / Fl-all % | 4.918 / 16.33 | 4.918 / 16.32 | 4.917 / 16.32 | |
120
+ | `gmflow-scale2-regrefine6-sintelft` | KITTI 2015 train, EPE / Fl-all % | 1.153 / 3.92 | 1.154 / 3.92 | 1.154 / 3.92 | trained on this dataset's public images; not a zero-shot result |
121
+ | `gmflow-scale2-regrefine6-kitti` | KITTI 2015 train, EPE / Fl-all % | 0.570 / 1.15 | 0.570 / 1.15 | 0.570 / 1.15 | trained on this dataset's public images; not a zero-shot result |
122
+ | `gmflow-scale2-regrefine6-mixdata` | KITTI 2015 train, EPE / Fl-all % | 1.315 / 4.64 | 1.315 / 4.63 | 1.315 / 4.64 | possibly in the training data |
123
+
124
+ Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks c4a67979, 2026-11-11.
125
+
126
+ ### Parity with the original code
127
+
128
+ Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0031 and p99 <= 0.021 (absolute error of the output `flow`).
129
+
130
+ ## Limitations
131
+
132
+ - No custom kernels backend is implemented.
133
+ - TensorRT builds at 1080x1920 fail for scale2 variants with a Myelin CHECK(is_const()) error on RTX 5060 Ti; scale1 variants build.
134
+
135
+ ## Usage
136
+
137
+ ```python
138
+ from fastforks import flow
139
+
140
+ model = flow.load("gmflow-scale1-mixdata") # mode="auto": the preferred backend from recorded measurements
141
+ result = model("frame1.png", "frame2.png")
142
+ result.flow # [2, H, W] float32 tensor: dx, dy in input pixels
143
+ result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
144
+ ```
145
+
146
+ Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `gmflow-scale1-mixdata`, `gmflow-scale1-things`, `gmflow-scale2-mixdata`, `gmflow-scale2-regrefine6-kitti`, `gmflow-scale2-regrefine6-mixdata`, `gmflow-scale2-regrefine6-sintelft`, `gmflow-scale2-regrefine6-things`, `gmflow-scale2-sintel`, `gmflow-scale2-things`.
147
+
148
+ ## Kernels backend
149
+
150
+ No kernels backend: after the fused attention portable is faster than the original and uses less memory at every tested size (e.g. scale1 1080x1920 556 vs 750 ms, 4496 vs 8360 MiB); the rest is the attention itself (55 % of the CUDA time at 1080p, already the memory-efficient / flash SDPA kernels) and float32 elementwise ops, and the convolutions are a small share, so no library in the manifests (cudnn_hacc speeds up convolutions) would change the total (measured 2026-11-15).
151
+
152
+ <!-- staged from fastforks 1cbf1671 (2026-11-15) -->