s2m2 / README.md

fastforks 0.1.0

f9f403b 9 days ago

8.76 kB

---
library_name: "fastforks"
pipeline_tag: "depth-estimation"
license: "cc-by-nc-4.0"
tags:
- "stereo"
- "stereo-matching"
- "disparity"
- "cuda"
- "pytorch"
- "fastforks"
---
# S2M2 (fastforks)

S2M2 (Min et al. 2025): multi-resolution transformer with optimal-transport matching; also outputs confidence and occlusion maps.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/junhong-3dv/s2m2> at `8f3bd785df59`.

Author weight repos: <https://huggingface.co/minimok/s2m2>.

Task: stereo. Tier: stable. Outputs: `disparity`, `confidence`, `occlusion`. Backends: `portable`, `kernels`.

Input: a rectified left / right image pair. Output: a disparity map (pixels, left view).

## Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/stereo/models/s2m2/card.toml).

| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
| `s2m2-l@1` | [CH256NTR3.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH256NTR3.pth) | `770e82227258` | CC-BY-NC-4.0 | **no** |
| `s2m2-m@1` | [CH192NTR2.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH192NTR2.pth) | `6e767e0977dd` | CC-BY-NC-4.0 | **no** |
| `s2m2-s@1` | [CH128NTR1.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH128NTR1.pth) | `93f7177d8159` | CC-BY-NC-4.0 | **no** |
| `s2m2-xl@1` | [CH384NTR3.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH384NTR3.pth) | `78dee08652ac` | CC-BY-NC-4.0 | **no** |

## Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: CC-BY-NC-4.0. Files: `LICENSE.md`.

Training data: README: extended training data incl. transparent objects.

Licence evidence: <https://github.com/junhong-3dv/s2m2/blob/8f3bd785df593619d955203c6f2c741bef6e31df/README.md#L24>

## Input size

any H x W >= 32, padded to a multiple of 32
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `s2m2-l` 1988x2880 (11.4 GiB peak allocated, 14.2 reserved); `s2m2-m` 1988x2880 (8.9 GiB peak allocated, 11.5 reserved); `s2m2-s` 1988x2880 (7.4 GiB peak allocated, 10.6 reserved); `s2m2-xl` 1024x2048 (7.3 GiB peak allocated, 8.5 reserved)

## Speed and memory

### Stereo, Middlebury 2014 half, 994x1482

| model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `s2m2-l` | 1936 | – | 1015 | 376 | – | 2.7x | – | 3245 / 3838 | 1249 / 1568 |
| `s2m2-m` | 1242 | 666 | 670 | 248 | 2.7x | 2.7x | 2371 / 2920 | 2300 / 2786 | 876 / 1162 |
| `s2m2-s` | 582 | 350 | 348 | 113 | 3.1x | 3.1x | 1759 / 1986 | 1733 / 1930 | 686 / 820 |
| `s2m2-xl` | 3510 | 1796 | 1815 | 682 | 2.6x | 2.7x | 5606 / 6080 | 5228 / 6016 | 2092 / 2274 |

### Startup and first call

| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- |
| `s2m2-l` | 1.22 (375x1242) | 0.99 | 0.58† | 0.67 (375x1242) | 1.01† | 0.37† |
| `s2m2-m` | 3.95 (375x1242) | 0.50 | 0.31† | 0.27 (375x1242) | 0.68† | 0.25† |
| `s2m2-s` | 0.22 (375x1242) | 0.22 | 0.20† | 0.14 (375x1242) | 0.35† | 0.11† |
| `s2m2-xl` | 4.50 (375x1242) | 2.73 | 0.90† | 0.53 (375x1242) | 1.81† | 0.68† |

### torch.compile and TensorRT

| model | input | portable ms | compile ms | compile s (cold) | TensorRT fp16 ms | ONNX + build s | kernels ms | peak MiB (alloc / reserved): portable | compile | TensorRT (device) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `s2m2-s` | 994x1482 | 348 | 185.9 | 95.9 | 137.1 | 363 | 113 | 1733 / 1930 | 1579 / 1710 | 980 |

### Fixed-size speed and peak memory, all models

| model | mode | 480x640 | 720x1280 | 1080x1920 | 12 MP 3000x4000 |
| --- | --- | --- | --- | --- | --- |
| `s2m2-l` | portable | 182.7 ms / 1304 / 1460 | 618.6 ms / 2499 / 2956 | 1459.4 ms / 4660 / 5680 | OOM |
| `s2m2-l` | kernels | 83.3 ms / 719 / 782 | 228.8 ms / 988 / 1224 | 534.8 ms / 1530 / 2250 | 4189.4 ms (n=3) / 6355 / 9246 |
| `s2m2-m` | portable | 114.5 ms / 780 / 904 | 402.5 ms / 1690 / 1954 | 973.5 ms / 3330 / 3896 | OOM |
| `s2m2-m` | kernels | 55.3 ms / 395 / 492 | 153.2 ms / 655 / 940 | 355.5 ms / 1119 / 1760 | 2793.3 ms (n=3) / 5151 / 7876 |
| `s2m2-s` | portable | 56.0 ms / 456 / 536 | 202.3 ms / 1141 / 1450 | 511.1 ms / 2427 / 3054 | OOM |
| `s2m2-s` | kernels | 32.7 ms / 212 / 294 | 70.2 ms / 452 / 692 | 161.9 ms / 887 / 1534 | 1203.3 ms / 4637 / 7166 |
| `s2m2-xl` | portable | 338.0 ms / 2476 / 2710 | 1112.0 ms / 4248 / 4842 | 2594.0 ms (n=3) / 7447 / 8684 | SKIPPED |
| `s2m2-xl` | kernels | 140.8 ms / 1441 / 1644 | 414.8 ms / 1748 / 2040 | 956.5 ms / 2493 / 3446 | 6933.0 ms (n=3) / 9200 / 11578 |

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the authors' code at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
## Accuracy
### On public datasets
Original = the authors' code, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
| `s2m2-s` | ETH3D two-view train, bad > 1 px % / EPE | 1.10 / 0.193 | 1.07 / 0.198 | 1.12 / 0.193 | 1.10 / 0.193 | possibly in the training data |
| `s2m2-s` | Middlebury 2014 trainingH, bad > 2 px % / EPE | 3.23 / 0.652 | 3.23 / 0.657 | 3.24 / 0.652 | 3.24 / 0.652 | possibly in the training data |
| `s2m2-m` | ETH3D two-view train, bad > 1 px % / EPE | 0.94 / 0.178 | 0.94 / 0.188 | 0.95 / 0.178 | 0.95 / 0.178 | possibly in the training data |
| `s2m2-m` | Middlebury 2014 trainingH, bad > 2 px % / EPE | 3.28 / 0.651 | 3.28 / 0.656 | 3.29 / 0.651 | 3.28 / 0.651 | possibly in the training data |
| `s2m2-l` | ETH3D two-view train, bad > 1 px % / EPE | 0.84 / 0.169 | 0.85 / 0.182 | 0.84 / 0.169 | 0.82 / 0.169 | possibly in the training data |
| `s2m2-l` | Middlebury 2014 trainingH, bad > 2 px % / EPE | 2.88 / 0.542 | 2.84 / 0.551 | 2.84 / 0.542 | 2.85 / 0.542 | possibly in the training data |
| `s2m2-xl` | ETH3D two-view train, bad > 1 px % / EPE | 0.53 / 0.147 | 0.52 / 0.157 | 0.53 / 0.147 | 0.52 / 0.149 | possibly in the training data |
| `s2m2-xl` | Middlebury 2014 trainingH, bad > 2 px % / EPE | 2.34 / 0.498 | 2.33 / 0.504 | 2.33 / 0.498 | 2.33 / 0.499 | possibly in the training data |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-05.
### Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.02 px and p99 <= 0.25 px.
## Limitations
- At 994x1482 portable S2M2-S is about as fast as the original code and allocates slightly more memory; use kernels for speed and memory.
## Usage
```python
from fastforks import stereo
# non-commercial weights: run with FASTFORKS_NONCOMMERCIAL=1 in the environment
model = stereo.load("s2m2-l")  # mode="auto": the preferred backend from recorded measurements
result = model("left.png", "right.png")
result.disparity  # [H, W] float32 tensor, input resolution
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
```
Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `s2m2-l`, `s2m2-m`, `s2m2-s`, `s2m2-xl`.
## Kernels backend
`stereo.load("s2m2-l", mode="kernels")` runs our CUDA kernels (`s2m2`, `s2m2_cutlass`, `cudnn_hacc`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
<!-- staged from fastforks feccac41 (2026-11-06) -->

	---
	library_name: "fastforks"
	pipeline_tag: "depth-estimation"
	license: "cc-by-nc-4.0"
	tags:
	- "stereo"
	- "stereo-matching"
	- "disparity"
	- "cuda"
	- "pytorch"
	- "fastforks"
	---
	# S2M2 (fastforks)

	S2M2 (Min et al. 2025): multi-resolution transformer with optimal-transport matching; also outputs confidence and occlusion maps.

	This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/junhong-3dv/s2m2> at `8f3bd785df59`.

	Author weight repos: <https://huggingface.co/minimok/s2m2>.

	Task: stereo. Tier: stable. Outputs: `disparity`, `confidence`, `occlusion`. Backends: `portable`, `kernels`.

	Input: a rectified left / right image pair. Output: a disparity map (pixels, left view).

	## Variants

	Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/stereo/models/s2m2/card.toml).

	\| variant \| weights (author link) \| SHA-256 \| weights licence \| commercial use \|
	\|---\|---\|---\|---\|---\|
	\| `s2m2-l@1` \| [CH256NTR3.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH256NTR3.pth) \| `770e82227258` \| CC-BY-NC-4.0 \| no \|
	\| `s2m2-m@1` \| [CH192NTR2.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH192NTR2.pth) \| `6e767e0977dd` \| CC-BY-NC-4.0 \| no \|
	\| `s2m2-s@1` \| [CH128NTR1.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH128NTR1.pth) \| `93f7177d8159` \| CC-BY-NC-4.0 \| no \|
	\| `s2m2-xl@1` \| [CH384NTR3.pth](https://huggingface.co/minimok/s2m2/resolve/880bc49f2a7cf54a42570f8550c46cb57454bac9/CH384NTR3.pth) \| `78dee08652ac` \| CC-BY-NC-4.0 \| no \|

	## Licence

	Original-code strings, recorded as they are (research-use package; check them before any other use). Code: CC-BY-NC-4.0. Files: `LICENSE.md`.

	Training data: README: extended training data incl. transparent objects.

	Licence evidence: <https://github.com/junhong-3dv/s2m2/blob/8f3bd785df593619d955203c6f2c741bef6e31df/README.md#L24>

	## Input size

	any H x W >= 32, padded to a multiple of 32
	Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `s2m2-l` 1988x2880 (11.4 GiB peak allocated, 14.2 reserved); `s2m2-m` 1988x2880 (8.9 GiB peak allocated, 11.5 reserved); `s2m2-s` 1988x2880 (7.4 GiB peak allocated, 10.6 reserved); `s2m2-xl` 1024x2048 (7.3 GiB peak allocated, 8.5 reserved)

	## Speed and memory

	### Stereo, Middlebury 2014 half, 994x1482

	\| model \| time (ms): original fp32 \| original \| portable \| kernels \| kernels speed-up: vs original \| vs portable \| peak memory (MiB, allocated / reserved): original \| portable \| kernels \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| `s2m2-l` \| 1936 \| – \| 1015 \| 376 \| – \| 2.7x \| – \| 3245 / 3838 \| 1249 / 1568 \|
	\| `s2m2-m` \| 1242 \| 666 \| 670 \| 248 \| 2.7x \| 2.7x \| 2371 / 2920 \| 2300 / 2786 \| 876 / 1162 \|
	\| `s2m2-s` \| 582 \| 350 \| 348 \| 113 \| 3.1x \| 3.1x \| 1759 / 1986 \| 1733 / 1930 \| 686 / 820 \|
	\| `s2m2-xl` \| 3510 \| 1796 \| 1815 \| 682 \| 2.6x \| 2.7x \| 5606 / 6080 \| 5228 / 6016 \| 2092 / 2274 \|

	### Startup and first call

	\| model \| load (s): original \| portable \| kernels \| first call (s): original \| portable \| kernels \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| `s2m2-l` \| 1.22 (375x1242) \| 0.99 \| 0.58† \| 0.67 (375x1242) \| 1.01† \| 0.37† \|
	\| `s2m2-m` \| 3.95 (375x1242) \| 0.50 \| 0.31† \| 0.27 (375x1242) \| 0.68† \| 0.25† \|
	\| `s2m2-s` \| 0.22 (375x1242) \| 0.22 \| 0.20† \| 0.14 (375x1242) \| 0.35† \| 0.11† \|
	\| `s2m2-xl` \| 4.50 (375x1242) \| 2.73 \| 0.90† \| 0.53 (375x1242) \| 1.81† \| 0.68† \|

	### torch.compile and TensorRT

	\| model \| input \| portable ms \| compile ms \| compile s (cold) \| TensorRT fp16 ms \| ONNX + build s \| kernels ms \| peak MiB (alloc / reserved): portable \| compile \| TensorRT (device) \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| `s2m2-s` \| 994x1482 \| 348 \| 185.9 \| 95.9 \| 137.1 \| 363 \| 113 \| 1733 / 1930 \| 1579 / 1710 \| 980 \|

	### Fixed-size speed and peak memory, all models

	\| model \| mode \| 480x640 \| 720x1280 \| 1080x1920 \| 12 MP 3000x4000 \|
	\| --- \| --- \| --- \| --- \| --- \| --- \|
	\| `s2m2-l` \| portable \| 182.7 ms / 1304 / 1460 \| 618.6 ms / 2499 / 2956 \| 1459.4 ms / 4660 / 5680 \| OOM \|
	\| `s2m2-l` \| kernels \| 83.3 ms / 719 / 782 \| 228.8 ms / 988 / 1224 \| 534.8 ms / 1530 / 2250 \| 4189.4 ms (n=3) / 6355 / 9246 \|
	\| `s2m2-m` \| portable \| 114.5 ms / 780 / 904 \| 402.5 ms / 1690 / 1954 \| 973.5 ms / 3330 / 3896 \| OOM \|
	\| `s2m2-m` \| kernels \| 55.3 ms / 395 / 492 \| 153.2 ms / 655 / 940 \| 355.5 ms / 1119 / 1760 \| 2793.3 ms (n=3) / 5151 / 7876 \|
	\| `s2m2-s` \| portable \| 56.0 ms / 456 / 536 \| 202.3 ms / 1141 / 1450 \| 511.1 ms / 2427 / 3054 \| OOM \|
	\| `s2m2-s` \| kernels \| 32.7 ms / 212 / 294 \| 70.2 ms / 452 / 692 \| 161.9 ms / 887 / 1534 \| 1203.3 ms / 4637 / 7166 \|
	\| `s2m2-xl` \| portable \| 338.0 ms / 2476 / 2710 \| 1112.0 ms / 4248 / 4842 \| 2594.0 ms (n=3) / 7447 / 8684 \| SKIPPED \|
	\| `s2m2-xl` \| kernels \| 140.8 ms / 1441 / 1644 \| 414.8 ms / 1748 / 2040 \| 956.5 ms / 2493 / 3446 \| 6933.0 ms (n=3) / 9200 / 11578 \|

	Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

	Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the authors' code at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

	## Accuracy

	### On public datasets

	Original = the authors' code, same checkpoint and protocol.

	\| variant \| dataset \| original fp32 \| original fp16 \| portable \| kernels \| note \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| `s2m2-s` \| ETH3D two-view train, bad > 1 px % / EPE \| 1.10 / 0.193 \| 1.07 / 0.198 \| 1.12 / 0.193 \| 1.10 / 0.193 \| possibly in the training data \|
	\| `s2m2-s` \| Middlebury 2014 trainingH, bad > 2 px % / EPE \| 3.23 / 0.652 \| 3.23 / 0.657 \| 3.24 / 0.652 \| 3.24 / 0.652 \| possibly in the training data \|
	\| `s2m2-m` \| ETH3D two-view train, bad > 1 px % / EPE \| 0.94 / 0.178 \| 0.94 / 0.188 \| 0.95 / 0.178 \| 0.95 / 0.178 \| possibly in the training data \|
	\| `s2m2-m` \| Middlebury 2014 trainingH, bad > 2 px % / EPE \| 3.28 / 0.651 \| 3.28 / 0.656 \| 3.29 / 0.651 \| 3.28 / 0.651 \| possibly in the training data \|
	\| `s2m2-l` \| ETH3D two-view train, bad > 1 px % / EPE \| 0.84 / 0.169 \| 0.85 / 0.182 \| 0.84 / 0.169 \| 0.82 / 0.169 \| possibly in the training data \|
	\| `s2m2-l` \| Middlebury 2014 trainingH, bad > 2 px % / EPE \| 2.88 / 0.542 \| 2.84 / 0.551 \| 2.84 / 0.542 \| 2.85 / 0.542 \| possibly in the training data \|
	\| `s2m2-xl` \| ETH3D two-view train, bad > 1 px % / EPE \| 0.53 / 0.147 \| 0.52 / 0.157 \| 0.53 / 0.147 \| 0.52 / 0.149 \| possibly in the training data \|
	\| `s2m2-xl` \| Middlebury 2014 trainingH, bad > 2 px % / EPE \| 2.34 / 0.498 \| 2.33 / 0.504 \| 2.33 / 0.498 \| 2.33 / 0.499 \| possibly in the training data \|

	Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-05.

	### Parity with the original code

	Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max \|diff\| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean \|diff\| <= 0.02 px and p99 <= 0.25 px.

	## Limitations

	- At 994x1482 portable S2M2-S is about as fast as the original code and allocates slightly more memory; use kernels for speed and memory.

	## Usage

	```python
	from fastforks import stereo

	# non-commercial weights: run with FASTFORKS_NONCOMMERCIAL=1 in the environment
	model = stereo.load("s2m2-l") # mode="auto": the preferred backend from recorded measurements
	result = model("left.png", "right.png")
	result.disparity # [H, W] float32 tensor, input resolution
	result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
	```

	Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `s2m2-l`, `s2m2-m`, `s2m2-s`, `s2m2-xl`.

	## Kernels backend

	`stereo.load("s2m2-l", mode="kernels")` runs our CUDA kernels (`s2m2`, `s2m2_cutlass`, `cudnn_hacc`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.

	<!-- staged from fastforks feccac41 (2026-11-06) -->