gmflow / README.md
Flor's picture
card: update from fastforks 1cbf1671
d9cf2e0 verified
|
Raw History Blame Contribute Delete
13.8 kB
---
library_name: "fastforks"
pipeline_tag: "image-to-image"
license: "other"
license_name: "see-model-card"
license_link: "https://github.com/autonomousvision/unimatch/blob/943a99006950ab6c84ea72a72beb147fdd03c19a/MODEL_ZOO.md"
tags:
- "flow"
- "optical-flow"
- "motion-estimation"
- "cuda"
- "pytorch"
- "fastforks"
- "fastforks-card"
---
# GMFlow (fastforks)
GMFlow (Xu, Zhang, Cai, Rezatofighi & Tao 2022) in its UniMatch release (Xu et al. 2023): optical flow as global matching. A CNN gives features at 1/8 resolution, a Transformer with self- and cross-attention (shifted windows) makes the two images' features aware of each other, the flow is the expected position of the best match over the whole image, and a self-attention layer spreads it into occluded areas. `scale2` variants repeat this at 1/4 resolution with local matching; `regrefine6` variants add 6 RAFT-like local refinements.
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/autonomousvision/unimatch> at `943a99006950`.
Task: flow. Tier: experimental. Outputs: `flow`. Backends: `portable`.
Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).
## Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/flow/models/gmflow/card.toml).
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
| `gmflow-scale1-mixdata@1` | [gmflow-scale1-mixdata-train320x576-4c3a6e9a.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale1-mixdata-train320x576-4c3a6e9a.pth) | `4c3a6e9af913` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale1-things@1` | [gmflow-scale1-things-e9887eda.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale1-things-e9887eda.pth) | `e9887eda6165` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-mixdata@1` | [gmflow-scale2-mixdata-train320x576-9ff1c094.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-mixdata-train320x576-9ff1c094.pth) | `9ff1c094f496` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-regrefine6-kitti@1` | [gmflow-scale2-regrefine6-kitti15-25b554d7.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-kitti15-25b554d7.pth) | `25b554d766e9` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-regrefine6-mixdata@1` | [gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth) | `4e7b215d5a25` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-regrefine6-sintelft@1` | [gmflow-scale2-regrefine6-sintelft-6e39e2b9.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-sintelft-6e39e2b9.pth) | `6e39e2b9fb0f` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-regrefine6-things@1` | [gmflow-scale2-regrefine6-things-776ed612.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-regrefine6-things-776ed612.pth) | `776ed612efb8` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-sintel@1` | [gmflow-scale2-sintel-3ed1cf48.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-sintel-3ed1cf48.pth) | `3ed1cf486cf7` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
| `gmflow-scale2-things@1` | [gmflow-scale2-things-36579974.pth](https://s3.eu-central-1.amazonaws.com/avg-projects/unimatch/pretrained/gmflow-scale2-things-36579974.pth) | `365799747d9f` | MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) | yes |
## Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: MIT AND Apache-2.0 AND BSD-3-Clause. Licence files: `LICENSE`, `LICENSE-Apache-2.0`, `LICENSE-RAFT`.
Code licence: MIT AND Apache-2.0 AND BSD-3-Clause; the authors do not state a separate weights licence.
Training data: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa) (FlyingChairs, then FlyingThings3D (part of SceneFlow)).
Licence evidence: <https://github.com/autonomousvision/unimatch/blob/943a99006950ab6c84ea72a72beb147fdd03c19a/MODEL_ZOO.md>
## Input size
- `gmflow-scale1-mixdata`, `gmflow-scale1-things`: any H x W >= 64, padded to a multiple of 16
- `gmflow-scale2-mixdata`, `gmflow-scale2-regrefine6-kitti`, `gmflow-scale2-regrefine6-mixdata`, `gmflow-scale2-regrefine6-sintelft`, `gmflow-scale2-regrefine6-things`, `gmflow-scale2-sintel`, `gmflow-scale2-things`: any H x W >= 64, padded to a multiple of 32
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `gmflow-scale1-mixdata` 1024x2048 (5.4 GiB peak allocated, 6.0 reserved); `gmflow-scale1-things` 1024x2048 (5.4 GiB peak allocated, 6.0 reserved); `gmflow-scale2-mixdata` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-kitti` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-mixdata` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-sintelft` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-regrefine6-things` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-sintel` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); `gmflow-scale2-things` 1024x2048 (6.3 GiB peak allocated, 13.1 reserved)
## Speed and memory
### Flow, synthetic
| model | input | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `gmflow-scale1-mixdata` | 1080x1920 | 1246 | 750 | 556 | – | – | – | 8360 / 8448 | 4496 / 4642 | – |
| `gmflow-scale1-things` | 1080x1920 | 1245 | 750 | 556 | – | – | – | 8360 / 8468 | 4496 / 4642 | – |
| `gmflow-scale2-mixdata` | 1080x1920 | 2544 | 1608 | 1139 | – | – | – | 8425 / 8572 | 6017 / 6224 | – |
| `gmflow-scale2-regrefine6-kitti` | 1080x1920 | 3255 | 2188 | 1678 | – | – | – | 8537 / 8822 | 6080 / 6230 | – |
| `gmflow-scale2-regrefine6-mixdata` | 1080x1920 | 3254 | 2188 | 1675 | – | – | – | 8537 / 8822 | 6080 / 6250 | – |
| `gmflow-scale2-regrefine6-sintelft` | 1080x1920 | 3255 | 2184 | 1674 | – | – | – | 8537 / 8822 | 6080 / 6250 | – |
| `gmflow-scale2-regrefine6-things` | 1080x1920 | 3230 | 2175 | 1665 | – | – | – | 8537 / 8822 | 6080 / 6230 | – |
| `gmflow-scale2-sintel` | 1080x1920 | 2541 | 1607 | 1139 | – | – | – | 8425 / 8552 | 6017 / 6224 | – |
| `gmflow-scale2-things` | 1080x1920 | 2537 | 1599 | 1139 | – | – | – | 8425 / 8552 | 6017 / 6224 | – |
### Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- |
| `gmflow-scale1-mixdata` | 0.13 | 0.16 | – | 0.76 | 0.56 | – |
| `gmflow-scale1-things` | 0.14 | 0.16 | – | 0.94 | 0.56 | – |
| `gmflow-scale2-mixdata` | 0.13 | 0.16 | – | 1.62 | 1.14 | – |
| `gmflow-scale2-regrefine6-kitti` | 0.15 | 0.46 | – | 2.20 | 1.68 | – |
| `gmflow-scale2-regrefine6-mixdata` | 0.14 | 0.25 | – | 2.20 | 1.68 | – |
| `gmflow-scale2-regrefine6-sintelft` | 0.15 | 0.26 | – | 2.37 | 1.68 | – |
| `gmflow-scale2-regrefine6-things` | 0.15 | 0.16 | – | 2.19 | 1.67 | – |
| `gmflow-scale2-things` | 0.14 | 0.17 | – | 1.61 | 1.14 | – |
### torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `gmflow-scale1-mixdata` | 1080x1920 | 556 ms | 382 ms | 187 ms | – | 29.1 s | 67.3 s | 4496 / 4642 | 4293 / 4558 | – | 4333 / 4390 |
| `gmflow-scale1-things` | 1080x1920 | 556 ms | 382 ms | 187 ms | – | 35.3 s | 69.3 s | 4496 / 4642 | 4293 / 4558 | – | 4333 / 4390 |
| `gmflow-scale2-mixdata` | 1080x1920 | 1139 ms | 765 ms | planned for 0.2.1 | – | 46.6 s | – | 6017 / 6224 | 10865 / 10904 | – | – |
| `gmflow-scale2-regrefine6-kitti` | 1080x1920 | 1678 ms | 1116 ms | planned for 0.2.1 | – | 58.4 s | – | 6080 / 6230 | 10875 / 10972 | – | – |
| `gmflow-scale2-regrefine6-mixdata` | 1080x1920 | 1675 ms | 1116 ms | planned for 0.2.1 | – | 58.9 s | – | 6080 / 6250 | 10875 / 10972 | – | – |
| `gmflow-scale2-regrefine6-sintelft` | 1080x1920 | 1674 ms | 1116 ms | planned for 0.2.1 | – | 58.5 s | – | 6080 / 6250 | 10875 / 10972 | – | – |
| `gmflow-scale2-regrefine6-things` | 1080x1920 | 1665 ms | 1116 ms | planned for 0.2.1 | – | 58.2 s | – | 6080 / 6230 | 10875 / 10972 | – | – |
| `gmflow-scale2-things` | 1080x1920 | 1139 ms | 765 ms | planned for 0.2.1 | – | 46.7 s | – | 6017 / 6224 | 10865 / 10904 | – | – |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
- `gmflow-scale2-sintel`: same network and settings as `gmflow-scale2-mixdata`, measured once (rows not repeated).
Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
## Accuracy
### On public datasets
Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | note |
|---|---|---|---|---|---|
| `gmflow-scale1-things` | KITTI 2015 train, EPE / Fl-all % | 10.347 / 34.31 | 10.347 / 34.30 | 10.346 / 34.30 | |
| `gmflow-scale1-mixdata` | KITTI 2015 train, EPE / Fl-all % | 3.698 / 17.60 | 3.698 / 17.61 | 3.697 / 17.61 | possibly in the training data |
| `gmflow-scale2-things` | KITTI 2015 train, EPE / Fl-all % | 6.747 / 22.09 | 6.747 / 22.09 | 6.747 / 22.09 | |
| `gmflow-scale2-sintel` | KITTI 2015 train, EPE / Fl-all % | 1.975 / 7.88 | 1.976 / 7.88 | 1.975 / 7.88 | trained on this dataset's public images; not a zero-shot result |
| `gmflow-scale2-mixdata` | KITTI 2015 train, EPE / Fl-all % | 2.390 / 10.10 | 2.390 / 10.10 | 2.389 / 10.10 | possibly in the training data |
| `gmflow-scale2-regrefine6-things` | KITTI 2015 train, EPE / Fl-all % | 4.918 / 16.33 | 4.918 / 16.32 | 4.917 / 16.32 | |
| `gmflow-scale2-regrefine6-sintelft` | KITTI 2015 train, EPE / Fl-all % | 1.153 / 3.92 | 1.154 / 3.92 | 1.154 / 3.92 | trained on this dataset's public images; not a zero-shot result |
| `gmflow-scale2-regrefine6-kitti` | KITTI 2015 train, EPE / Fl-all % | 0.570 / 1.15 | 0.570 / 1.15 | 0.570 / 1.15 | trained on this dataset's public images; not a zero-shot result |
| `gmflow-scale2-regrefine6-mixdata` | KITTI 2015 train, EPE / Fl-all % | 1.315 / 4.64 | 1.315 / 4.63 | 1.315 / 4.64 | possibly in the training data |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks c4a67979, 2026-11-11.
### Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0031 and p99 <= 0.021 (absolute error of the output `flow`).
## Limitations
- No custom kernels backend is implemented.
- TensorRT builds at 1080x1920 fail for scale2 variants with a Myelin CHECK(is_const()) error on RTX 5060 Ti; scale1 variants build.
## Usage
```python
from fastforks import flow
model = flow.load("gmflow-scale1-mixdata") # mode="auto": the preferred backend from recorded measurements
result = model("frame1.png", "frame2.png")
result.flow # [2, H, W] float32 tensor: dx, dy in input pixels
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
```
Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `gmflow-scale1-mixdata`, `gmflow-scale1-things`, `gmflow-scale2-mixdata`, `gmflow-scale2-regrefine6-kitti`, `gmflow-scale2-regrefine6-mixdata`, `gmflow-scale2-regrefine6-sintelft`, `gmflow-scale2-regrefine6-things`, `gmflow-scale2-sintel`, `gmflow-scale2-things`.
## Kernels backend
No kernels backend: after the fused attention portable is faster than the original and uses less memory at every tested size (e.g. scale1 1080x1920 556 vs 750 ms, 4496 vs 8360 MiB); the rest is the attention itself (55 % of the CUDA time at 1080p, already the memory-efficient / flash SDPA kernels) and float32 elementwise ops, and the convolutions are a small share, so no library in the manifests (cudnn_hacc speeds up convolutions) would change the total (measured 2026-11-15).
<!-- staged from fastforks 1cbf1671 (2026-11-15) -->