gmflow / README.md
Flor's picture
card: update from fastforks 1cbf1671
d9cf2e0 verified
|
Raw History Blame Contribute Delete
13.8 kB
metadata
library_name: fastforks
pipeline_tag: image-to-image
license: other
license_name: see-model-card
license_link: >-
  https://github.com/autonomousvision/unimatch/blob/943a99006950ab6c84ea72a72beb147fdd03c19a/MODEL_ZOO.md
tags:
  - flow
  - optical-flow
  - motion-estimation
  - cuda
  - pytorch
  - fastforks
  - fastforks-card

GMFlow (fastforks)

GMFlow (Xu, Zhang, Cai, Rezatofighi & Tao 2022) in its UniMatch release (Xu et al. 2023): optical flow as global matching. A CNN gives features at 1/8 resolution, a Transformer with self- and cross-attention (shifted windows) makes the two images' features aware of each other, the flow is the expected position of the best match over the whole image, and a self-attention layer spreads it into occluded areas. scale2 variants repeat this at 1/4 resolution with local matching; regrefine6 variants add 6 RAFT-like local refinements.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/autonomousvision/unimatch at 943a99006950.

Task: flow. Tier: experimental. Outputs: flow. Backends: portable.

Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
gmflow-scale1-mixdata@1 gmflow-scale1-mixdata-train320x576-4c3a6e9a.pth 4c3a6e9af913 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale1-things@1 gmflow-scale1-things-e9887eda.pth e9887eda6165 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-mixdata@1 gmflow-scale2-mixdata-train320x576-9ff1c094.pth 9ff1c094f496 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-regrefine6-kitti@1 gmflow-scale2-regrefine6-kitti15-25b554d7.pth 25b554d766e9 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-regrefine6-mixdata@1 gmflow-scale2-regrefine6-mixdata-train320x576-4e7b215d.pth 4e7b215d5a25 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-regrefine6-sintelft@1 gmflow-scale2-regrefine6-sintelft-6e39e2b9.pth 6e39e2b9fb0f MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-regrefine6-things@1 gmflow-scale2-regrefine6-things-776ed612.pth 776ed612efb8 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-sintel@1 gmflow-scale2-sintel-3ed1cf48.pth 3ed1cf486cf7 MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes
gmflow-scale2-things@1 gmflow-scale2-things-36579974.pth 365799747d9f MIT AND Apache-2.0 AND BSD-3-Clause (assumed from code) yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: MIT AND Apache-2.0 AND BSD-3-Clause. Licence files: LICENSE, LICENSE-Apache-2.0, LICENSE-RAFT.

Code licence: MIT AND Apache-2.0 AND BSD-3-Clause; the authors do not state a separate weights licence.

Training data: FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa) (FlyingChairs, then FlyingThings3D (part of SceneFlow)).

Licence evidence: https://github.com/autonomousvision/unimatch/blob/943a99006950ab6c84ea72a72beb147fdd03c19a/MODEL_ZOO.md

Input size

  • gmflow-scale1-mixdata, gmflow-scale1-things: any H x W >= 64, padded to a multiple of 16
  • gmflow-scale2-mixdata, gmflow-scale2-regrefine6-kitti, gmflow-scale2-regrefine6-mixdata, gmflow-scale2-regrefine6-sintelft, gmflow-scale2-regrefine6-things, gmflow-scale2-sintel, gmflow-scale2-things: any H x W >= 64, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): gmflow-scale1-mixdata 1024x2048 (5.4 GiB peak allocated, 6.0 reserved); gmflow-scale1-things 1024x2048 (5.4 GiB peak allocated, 6.0 reserved); gmflow-scale2-mixdata 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); gmflow-scale2-regrefine6-kitti 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); gmflow-scale2-regrefine6-mixdata 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); gmflow-scale2-regrefine6-sintelft 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); gmflow-scale2-regrefine6-things 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); gmflow-scale2-sintel 1024x2048 (6.3 GiB peak allocated, 13.1 reserved); gmflow-scale2-things 1024x2048 (6.3 GiB peak allocated, 13.1 reserved)

Speed and memory

Flow, synthetic

model input time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
gmflow-scale1-mixdata 1080x1920 1246 750 556 – – – 8360 / 8448 4496 / 4642 –
gmflow-scale1-things 1080x1920 1245 750 556 – – – 8360 / 8468 4496 / 4642 –
gmflow-scale2-mixdata 1080x1920 2544 1608 1139 – – – 8425 / 8572 6017 / 6224 –
gmflow-scale2-regrefine6-kitti 1080x1920 3255 2188 1678 – – – 8537 / 8822 6080 / 6230 –
gmflow-scale2-regrefine6-mixdata 1080x1920 3254 2188 1675 – – – 8537 / 8822 6080 / 6250 –
gmflow-scale2-regrefine6-sintelft 1080x1920 3255 2184 1674 – – – 8537 / 8822 6080 / 6250 –
gmflow-scale2-regrefine6-things 1080x1920 3230 2175 1665 – – – 8537 / 8822 6080 / 6230 –
gmflow-scale2-sintel 1080x1920 2541 1607 1139 – – – 8425 / 8552 6017 / 6224 –
gmflow-scale2-things 1080x1920 2537 1599 1139 – – – 8425 / 8552 6017 / 6224 –

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
gmflow-scale1-mixdata 0.13 0.16 – 0.76 0.56 –
gmflow-scale1-things 0.14 0.16 – 0.94 0.56 –
gmflow-scale2-mixdata 0.13 0.16 – 1.62 1.14 –
gmflow-scale2-regrefine6-kitti 0.15 0.46 – 2.20 1.68 –
gmflow-scale2-regrefine6-mixdata 0.14 0.25 – 2.20 1.68 –
gmflow-scale2-regrefine6-sintelft 0.15 0.26 – 2.37 1.68 –
gmflow-scale2-regrefine6-things 0.15 0.16 – 2.19 1.67 –
gmflow-scale2-things 0.14 0.17 – 1.61 1.14 –

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
gmflow-scale1-mixdata 1080x1920 556 ms 382 ms 187 ms – 29.1 s 67.3 s 4496 / 4642 4293 / 4558 – 4333 / 4390
gmflow-scale1-things 1080x1920 556 ms 382 ms 187 ms – 35.3 s 69.3 s 4496 / 4642 4293 / 4558 – 4333 / 4390
gmflow-scale2-mixdata 1080x1920 1139 ms 765 ms planned for 0.2.1 – 46.6 s – 6017 / 6224 10865 / 10904 – –
gmflow-scale2-regrefine6-kitti 1080x1920 1678 ms 1116 ms planned for 0.2.1 – 58.4 s – 6080 / 6230 10875 / 10972 – –
gmflow-scale2-regrefine6-mixdata 1080x1920 1675 ms 1116 ms planned for 0.2.1 – 58.9 s – 6080 / 6250 10875 / 10972 – –
gmflow-scale2-regrefine6-sintelft 1080x1920 1674 ms 1116 ms planned for 0.2.1 – 58.5 s – 6080 / 6250 10875 / 10972 – –
gmflow-scale2-regrefine6-things 1080x1920 1665 ms 1116 ms planned for 0.2.1 – 58.2 s – 6080 / 6230 10875 / 10972 – –
gmflow-scale2-things 1080x1920 1139 ms 765 ms planned for 0.2.1 – 46.7 s – 6017 / 6224 10865 / 10904 – –

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

  • gmflow-scale2-sintel: same network and settings as gmflow-scale2-mixdata, measured once (rows not repeated).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable note
gmflow-scale1-things KITTI 2015 train, EPE / Fl-all % 10.347 / 34.31 10.347 / 34.30 10.346 / 34.30
gmflow-scale1-mixdata KITTI 2015 train, EPE / Fl-all % 3.698 / 17.60 3.698 / 17.61 3.697 / 17.61 possibly in the training data
gmflow-scale2-things KITTI 2015 train, EPE / Fl-all % 6.747 / 22.09 6.747 / 22.09 6.747 / 22.09
gmflow-scale2-sintel KITTI 2015 train, EPE / Fl-all % 1.975 / 7.88 1.976 / 7.88 1.975 / 7.88 trained on this dataset's public images; not a zero-shot result
gmflow-scale2-mixdata KITTI 2015 train, EPE / Fl-all % 2.390 / 10.10 2.390 / 10.10 2.389 / 10.10 possibly in the training data
gmflow-scale2-regrefine6-things KITTI 2015 train, EPE / Fl-all % 4.918 / 16.33 4.918 / 16.32 4.917 / 16.32
gmflow-scale2-regrefine6-sintelft KITTI 2015 train, EPE / Fl-all % 1.153 / 3.92 1.154 / 3.92 1.154 / 3.92 trained on this dataset's public images; not a zero-shot result
gmflow-scale2-regrefine6-kitti KITTI 2015 train, EPE / Fl-all % 0.570 / 1.15 0.570 / 1.15 0.570 / 1.15 trained on this dataset's public images; not a zero-shot result
gmflow-scale2-regrefine6-mixdata KITTI 2015 train, EPE / Fl-all % 1.315 / 4.64 1.315 / 4.63 1.315 / 4.64 possibly in the training data

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks c4a67979, 2026-11-11.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0031 and p99 <= 0.021 (absolute error of the output flow).

Limitations

  • No custom kernels backend is implemented.
  • TensorRT builds at 1080x1920 fail for scale2 variants with a Myelin CHECK(is_const()) error on RTX 5060 Ti; scale1 variants build.

Usage

from fastforks import flow

model = flow.load("gmflow-scale1-mixdata")  # mode="auto": the preferred backend from recorded measurements
result = model("frame1.png", "frame2.png")
result.flow  # [2, H, W] float32 tensor: dx, dy in input pixels
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: gmflow-scale1-mixdata, gmflow-scale1-things, gmflow-scale2-mixdata, gmflow-scale2-regrefine6-kitti, gmflow-scale2-regrefine6-mixdata, gmflow-scale2-regrefine6-sintelft, gmflow-scale2-regrefine6-things, gmflow-scale2-sintel, gmflow-scale2-things.

Kernels backend

No kernels backend: after the fused attention portable is faster than the original and uses less memory at every tested size (e.g. scale1 1080x1920 556 vs 750 ms, 4496 vs 8360 MiB); the rest is the attention itself (55 % of the CUDA time at 1080p, already the memory-efficient / flash SDPA kernels) and float32 elementwise ops, and the convolutions are a small share, so no library in the manifests (cudnn_hacc speeds up convolutions) would change the total (measured 2026-11-15).