SEA-RAFT (fastforks)

SEA-RAFT (Wang, Lipson & Deng 2024): RAFT with ResNet encoders, an initial flow regressed from the context features and a ConvNeXt update block, so 4 iterations suffice; correlation pyramid at 1/8 resolution, convex upsampling. The authors' released weights (pretrained on TartanAir); the network is ported from princeton-vl/SEA-RAFT. sea-raft-m / sea-raft-s are the general checkpoints (trained on the FlyingThings3D, Sintel, KITTI and HD1K mix); the -spring variants are made for 1080p footage and work at half resolution, as upstream evaluates them: the pair is halved (bilinear), the flow is upsampled and doubled. Paper (zero-shot after FlyingChairs + FlyingThings, the C-T weights): KITTI 2015 train EPE 4.29 / Fl-all 14.2 % (M), Sintel train EPE 1.21 clean / 4.04 final (M); M with 12 iterations (iters=12) is the paper's SEA-RAFT (L).

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/princeton-vl/SEA-RAFT at 9137517ba24e.

Author weight repos: https://huggingface.co/MemorySlices/Tartan-C-T-TSKH-kitti432x960-M, https://huggingface.co/MemorySlices/Tartan-C-T-TSKH-kitti432x960-S, https://huggingface.co/MemorySlices/Tartan-C-T-TSKH-spring540x960-M, https://huggingface.co/MemorySlices/Tartan-C-T-TSKH-spring540x960-S, https://huggingface.co/MemorySlices/Tartan-C-T-TSKH432x960-M, https://huggingface.co/MemorySlices/Tartan-C-T-TSKH432x960-S, https://huggingface.co/MemorySlices/Tartan-C-T432x960-M.

Task: flow. Tier: stable. Outputs: flow. Backends: portable, kernels.

Input: two RGB images (consecutive frames). Output: per-pixel motion from the first to the second (dx, dy in pixels).

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
sea-raft-m@1 Tartan-C-T-TSKH432x960-M.safetensors 83e4319d1029 BSD-3-Clause yes
sea-raft-m-kitti@1 Tartan-C-T-TSKH-kitti432x960-M.safetensors 9a2d2a9b9e78 BSD-3-Clause yes
sea-raft-m-spring@1 Tartan-C-T-TSKH-spring540x960-M.safetensors cb8cfbf14c5e BSD-3-Clause yes
sea-raft-m-things@1 Tartan-C-T432x960-M.safetensors 41f9a087377a BSD-3-Clause yes
sea-raft-s@1 Tartan-C-T-TSKH432x960-S.safetensors 9752a49fca0a BSD-3-Clause yes
sea-raft-s-kitti@1 Tartan-C-T-TSKH-kitti432x960-S.safetensors b607b2ca2e55 BSD-3-Clause yes
sea-raft-s-spring@1 Tartan-C-T-TSKH-spring540x960-S.safetensors d6a75e47f263 BSD-3-Clause yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: BSD-3-Clause. Licence files: LICENSE.

Training data: TartanAir, FlyingChairs (Freiburg), Freiburg SceneFlow (FlyingThings3D, Driving, Monkaa), MPI Sintel optical flow, KITTI stereo / optical flow 2012 / 2015, HCI HD1K optical flow benchmark (TartanAir pretraining, FlyingChairs, FlyingThings3D (part of SceneFlow), then a mix of FlyingThings3D, Sintel, KITTI 2015 and HD1K (TSKH)).

Licence evidence: https://huggingface.co/MemorySlices/Tartan-C-T-TSKH432x960-M/blob/85a47dc9096cca6e907ef9100dc1847a19b26a2a/README.md

Input size

  • sea-raft-m, sea-raft-m-kitti, sea-raft-m-things, sea-raft-s, sea-raft-s-kitti: any H x W >= 128, padded to a multiple of 8
  • sea-raft-m-spring, sea-raft-s-spring: any H x W >= 256, padded to a multiple of 8 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): sea-raft-m 1024x2048 (8.3 GiB peak allocated, 9.4 reserved); sea-raft-m-kitti 1024x2048 (8.3 GiB peak allocated, 9.4 reserved); sea-raft-m-spring 1988x2880 (4.1 GiB peak allocated, 4.8 reserved); sea-raft-m-things 1024x2048 (8.3 GiB peak allocated, 9.4 reserved); sea-raft-s 1024x2048 (8.3 GiB peak allocated, 9.3 reserved); sea-raft-s-kitti 1024x2048 (8.3 GiB peak allocated, 9.3 reserved); sea-raft-s-spring 1988x2880 (4.1 GiB peak allocated, 4.8 reserved)

Speed and memory

Flow, synthetic

model input time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
sea-raft-m 1080x1920 504 367 270 73.1 5.0x 3.7x 7385 / 8570 6060 / 6180 435 / 514
sea-raft-m-spring 1080x1920 96.1 56.7 46.8 23.2 2.4x 2.0x 761 / 812 652 / 700 181 / 214
sea-raft-s 1080x1920 365 300 211 50.4 6.0x 4.2x 7323 / 8502 6019 / 6136 415 / 492
sea-raft-s-kitti 1080x1920 365 299 210 50.5 5.9x 4.2x 7323 / 8502 6019 / 6136 415 / 492
sea-raft-s-spring 1080x1920 67.1 42.4 32.8 16.0 2.6x 2.0x 699 / 764 611 / 656 160 / 192

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
sea-raft-m 0.53 0.29 1.15 0.37 0.27 0.07
sea-raft-m-kitti 0.67 1.02 1.14 0.37 0.27 0.07
sea-raft-m-spring 0.57 0.74 1.14 0.06 0.05 0.02
sea-raft-m-things 0.55 0.29 1.13 0.37 0.27 0.07
sea-raft-s 0.30 0.18 1.10 0.30 0.21 0.05
sea-raft-s-kitti 0.30 0.27 1.10 0.30 0.21 0.05
sea-raft-s-spring 0.31 0.34 1.11 0.04 0.04 0.02

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
sea-raft-m 1080x1920 270 ms 191 ms 132 ms 73.1 ms 34.3 s 70.0 s 6060 / 6180 5869 / 5920 435 / 514 11110 / 11130
sea-raft-m-kitti 1080x1920 270 ms 191 ms 132 ms 73.1 ms 34.9 s 69.7 s 6060 / 6180 5869 / 5920 435 / 514 11110 / 11130
sea-raft-m-spring 1080x1920 46.8 ms 34.6 ms 4.43 ms 23.2 ms 23.2 s 47.1 s 652 / 700 624 / 640 181 / 214 291 / 318
sea-raft-m-things 1080x1920 270 ms 192 ms 132 ms 73.1 ms 34.9 s 73.6 s 6060 / 6180 5869 / 5920 435 / 514 11110 / 11130
sea-raft-s 1080x1920 211 ms 150 ms 108 ms 50.4 ms 35.4 s 68.0 s 6019 / 6136 5828 / 5876 415 / 492 11043 / 11078
sea-raft-s-kitti 1080x1920 210 ms 150 ms 108 ms 50.5 ms 31.8 s 67.3 s 6019 / 6136 5828 / 5876 415 / 492 11043 / 11078
sea-raft-s-spring 1080x1920 32.8 ms 23.8 ms 3.01 ms 16.0 ms 21.9 s 38.8 s 611 / 656 582 / 596 160 / 192 215 / 278

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

  • sea-raft-m-kitti, sea-raft-m-things: same network and settings as sea-raft-m, measured once (rows not repeated).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Frames _10 -> _11 at full size against the flow_occ ground truth; EPE = mean of the per-pair mean end-point error, Fl-all = share of all valid pixels with an error > 3 px and > 5 % of the GT magnitude. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
sea-raft-m KITTI 2015 train, EPE / Fl-all % 0.880 / 2.51 0.880 / 2.51 0.880 / 2.51 0.880 / 2.50 trained on this dataset's public images; not a zero-shot result
sea-raft-m-kitti KITTI 2015 train, EPE / Fl-all % 0.808 / 2.16 0.808 / 2.16 0.808 / 2.16 0.808 / 2.16 trained on this dataset's public images; not a zero-shot result
sea-raft-m-spring KITTI 2015 train, EPE / Fl-all % 6.779 / 23.75 6.775 / 23.74 6.775 / 23.74 6.783 / 23.74 trained on this dataset's public images; not a zero-shot result
sea-raft-m-things KITTI 2015 train, EPE / Fl-all % 4.293 / 14.31 4.290 / 14.32 4.289 / 14.32 4.307 / 14.32
sea-raft-s KITTI 2015 train, EPE / Fl-all % 1.054 / 3.13 1.054 / 3.13 1.054 / 3.13 1.054 / 3.14 trained on this dataset's public images; not a zero-shot result
sea-raft-s-kitti KITTI 2015 train, EPE / Fl-all % 0.936 / 2.66 0.936 / 2.66 0.936 / 2.66 0.935 / 2.65 trained on this dataset's public images; not a zero-shot result
sea-raft-s-spring KITTI 2015 train, EPE / Fl-all % 7.758 / 26.52 7.758 / 26.52 7.758 / 26.52 7.757 / 26.51 trained on this dataset's public images; not a zero-shot result

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks f2992269+uncommitted, 2026-11-10.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.00025 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.002 and p99 <= 0.014 (absolute error of the output flow).

Usage

from fastforks import flow

model = flow.load("sea-raft-m")  # mode="auto": the preferred backend from recorded measurements
result = model("frame1.png", "frame2.png")
result.flow  # [2, H, W] float32 tensor: dx, dy in input pixels
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: sea-raft-m, sea-raft-m-kitti, sea-raft-m-spring, sea-raft-m-things, sea-raft-s, sea-raft-s-kitti, sea-raft-s-spring.

Kernels backend

flow.load("sea-raft-m", mode="kernels") runs our CUDA kernels (raft_flow, cutlass_conv, cudnn_hacc). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including fastforks/sea-raft