Add checkpoint manifest and configuration-based downloads

e5491c9 verified 2 days ago

9.11 kB

---
language:
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: depth-estimation
tags:
- computer-vision
- multi-view-reconstruction
- camera-pose-estimation
- point-tracking
- scene-flow
- image-matching
---
# Kosmo-GFM: A Production-Ready Geometry Foundation Model

**Kosmo Research**

[Paper](https://arxiv.org/abs/2611.16840) · [Project](https://kosmoresearch.github.io/Kosmo-GFM/) · [Code](https://github.com/KosmoResearch/Kosmo-GFM) · [Checkpoints](https://huggingface.co/KosmoResearch/Kosmo-GFM/tree/main)

![Kosmo-GFM: static scene reconstructions and dynamic 3D motion trajectories](assets/kosmo-gfm-overview.png)

## Overview

Kosmo-GFM predicts camera parameters, depth, cross-view correspondences, and
dynamic 3D points from images and videos. A shared multi-view encoder supports
task-specific geometry, motion, and matching decoders, with optional camera
conditioning for reconstruction from known poses.

The code repository provides standalone inference for **GFM**, **Camera Condition GFM**,
**GFM 4D**, and **MVLoMA**. The full PoseEngine and Gaussian Splatting pipelines
described in the technical report are not included.

## Model Cards

All geometry models use a ViT-g backbone. Parameter counts are rounded for the
provided checkpoints. ✅ indicates a supported capability; — indicates it is
not part of that inference configuration.

| Model | Params | Depth | Camera Est. | Camera Cond. | 4D Motion | Matching |
| --- | --- | --- | --- | --- | --- | --- |
| [GFM](#gfm) | 1.66B | ✅ | ✅ | — | — | — |
| [Camera Condition GFM](#camera-condition-gfm) | 1.77B | ✅ | ✅ | ✅ | — | — |
| [GFM 4D](#gfm-4d) | 1.65B | ✅ | ✅ | — | ✅ | — |
| [GFM + MVLoMA](#gfm-with-mvloma) | 1.73B | ✅ | ✅ | — | — | ✅ |

- **Camera Condition GFM** is the **Camera-pose scale** version and takes
  camera extrinsics and intrinsics as input.
- **GFM + MVLoMA** shares the GFM encoder and adds a 75.9M-parameter matching
  head. It requires both the GFM and MVLoMA checkpoints.
- **GFM 4D** predicts geometry and cross-frame motion for Dynamic Points and
  Scene Flow evaluation.

| Checkpoint | Inference configuration |
| --- | --- |
| [`gfm_3d.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/gfm_3d.pth) | GFM: depth, cameras, and point clouds |
| [`gfm_3d_camera_cond.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/gfm_3d_camera_cond.pth) | Camera Condition GFM: Camera-pose scale version |
| [`gfm_4d.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/gfm_4d.pth) | GFM 4D: geometry and cross-frame motion |
| [`mvloma.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/mvloma.pth) | MVLoMA matching head; requires `gfm_3d.pth` |

## Quick Start

Download the code and checkpoints, then run inference from the code repository root.

### Installation

Requires Python 3.10+ and a CUDA GPU. Install a compatible PyTorch / torchvision
build, then:

```bash
git clone https://github.com/KosmoResearch/Kosmo-GFM.git
cd Kosmo-GFM
pip install -e .
pip install -U huggingface_hub
# Optional PLY/GLB export
pip install -r requirements-optional.txt
```

### Download Checkpoints

Download all four checkpoints into the `weights/` directory. The checkpoint
manifest maps each inference configuration to its required weight files:

```python
import json
from pathlib import Path
from huggingface_hub import hf_hub_download, snapshot_download
repo_id = "KosmoResearch/Kosmo-GFM"
config_path = hf_hub_download(
    repo_id=repo_id, filename="config.json", local_dir="weights"
)
config = json.loads(Path(config_path).read_text())
selected_models = list(config["models"])  # All inference configurations.
# For GFM + MVLoMA only: selected_models = ["gfm_3d_with_mvloma"]
checkpoints = sorted({
    filename
    for name in selected_models
    for filename in config["models"][name]["checkpoints"]
})
snapshot_download(
    repo_id=repo_id, local_dir="weights", allow_patterns=checkpoints
)
```

To download a subset, set `selected_models` to any of `gfm_3d`,
`gfm_3d_camera_cond`, `gfm_4d`, or `gfm_3d_with_mvloma`.
The MVLoMA configuration includes both `gfm_3d.pth` and `mvloma.pth`.
[`config.json`](config.json) is a checkpoint manifest for the standalone GFM
scripts; it is not a Transformers `AutoModel` configuration.

See the [Hugging Face download guide](https://huggingface.co/docs/huggingface_hub/en/guides/download)
for authentication and download options.

These are PyTorch checkpoints loaded by the GFM inference scripts.

### Inference

Run from the repository root with your own images or video frames.

#### GFM

```bash
python tools/infer_3d.py \
  --checkpoint weights/gfm_3d.pth --input /path/to/images \
  --output outputs/gfm --save-npz --no-ply --no-glb
```

Exports depth, confidence, cameras and point clouds to `predictions.npz`.
The default backbone is `vitg`; use `--model-size` to match your checkpoint.

#### Camera Condition GFM

The provided checkpoint is the **Camera-pose scale** version.

```bash
python tools/infer_3d_camera_cond.py \
  --checkpoint weights/gfm_3d_camera_cond.pth --input /path/to/images \
  --camera-npz /path/to/cameras.npz --output outputs/gfm_camera \
  --save-npz --no-ply --no-glb
```

The camera NPZ requires `extrinsics` (`[N,4,4]` or `[N,3,4]`, OpenCV
world-to-camera) and `intrinsics` (`[N,3,3]`, at original image resolution).
Cameras follow filename-sorted image order unless `image_names` is supplied.
Use `--camera-convention c2w` for camera-to-world matrices.

#### GFM 4D

```bash
python tools/infer_4d.py \
  --checkpoint weights/gfm_4d.pth --input /path/to/frames \
  --output outputs/gfm_4d --save-npz --no-ply --no-glb
```

Exports geometry, cameras and motion to `predictions.npz` at relative scene scale.
Use `--chunk-size` to control query decoding memory.

#### GFM with MVLoMA

```bash
python tools/infer_3d_with_mvloma.py \
  --checkpoint weights/gfm_3d.pth --mvloma-checkpoint weights/mvloma.pth \
  --input /path/to/images --output outputs/gfm_with_mvloma --no-ply --no-glb
```

Exports geometry to `predictions.npz` and image matches to `matches.npz`.
Requires at least two images. Use `--all-pairs` to export every image pair.

For PLY/GLB output, install the optional dependencies and omit
`--no-ply --no-glb`. Each script provides additional options through `--help`.

## Benchmarks

### SpatialBench

GFM results on SpatialBench.
Higher is better for ↑; lower is better for ↓. Accuracy and AUC values are
reported on a 0–1 scale.

**Depth**

| Setting | AbsRel↓ | SqRel↓ | RMSE↓ | δ1.03↑ | δ1.05↑ | δ1.10↑ |
| --- | --- | --- | --- | --- | --- | --- |
| Sparse | 0.0630 | 0.1093 | 0.6207 | 0.6696 | 0.7810 | 0.8816 |
| Medium | 0.0594 | 0.0884 | 0.5822 | 0.6895 | 0.8077 | 0.9025 |

**Camera**

| Setting | RAcc3↑ | RAcc5↑ | TAcc3↑ | TAcc5↑ | AUC@5↑ | AUC@15↑ | AUC@30↑ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Sparse | 0.8358 | 0.9055 | 0.6457 | 0.7394 | 0.5726 | 0.7412 | 0.8226 |
| Medium | 0.8310 | 0.8938 | 0.6468 | 0.7322 | 0.5820 | 0.7450 | 0.8281 |

See [SpatialBench](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/bench/SpatialBench/README.md) for evaluation usage.

### GFM4DBench

GFM results on GFM4DBench.

| Dataset | Point APD (%)↑ | Point EPE (m)↓ | Flow Accuracy (%)↑ | Flow EPE (m)↓ |
| --- | --- | --- | --- | --- |
| Panoptic Studio | 83.41 | 0.1411 | 50.15 | 0.1257 |
| PointOdyssey | 85.73 | 0.1474 | 83.02 | 0.0521 |
| Dynamic Replica | 80.76 | 0.2017 | 91.31 | 0.0279 |
| Aria Digital Twin | 89.78 | 0.1102 | 96.34 | 0.0134 |
| SynthVerse | 42.49 | 1.1105 | 34.56 | 0.7213 |
| Mean | 76.43 | 0.3422 | 71.08 | 0.1881 |

Mean is the arithmetic average across all five datasets.

See [GFM4DBench](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/bench/GFM4DBench/README.md) for Dynamic Points and Scene Flow
evaluation on TapVid3D / WorldTrack and SynthVerse.

## Acknowledgements

Thanks to [VGGT](https://github.com/facebookresearch/vggt),
[DINOv2](https://github.com/facebookresearch/dinov2),
[Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3),
[DPT](https://github.com/isl-org/DPT), [DeDoDe](https://github.com/Parskatt/DeDoDe),
[DaD](https://github.com/Parskatt/DaD), [LightGlue](https://github.com/cvg/LightGlue),
[SpatialBench](https://github.com/Ropedia/SpatialBench), and
[Open-d4rt](https://github.com/Lijiaxin0111/Open-d4rt).

## Citation

```bibtex
@techreport{kosmo2026gfm,
  title={{Kosmo-GFM}: A Production-Ready Geometry Foundation Model},
  author={{Kosmo Research}},
  institution={Kosmo Research},
  year={2026},
  url={https://arxiv.org/abs/2611.16840}
}
```

## License

Original code is licensed under [Apache-2.0](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/LICENSE). Third-party terms,
including the [VGGT use restrictions](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/licenses/VGGT-LICENSE.txt), also apply;
see [NOTICE](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/NOTICE) and [provenance notes](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/licenses/README.md).

	---
	language:
	- en
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: depth-estimation
	tags:
	- computer-vision
	- multi-view-reconstruction
	- camera-pose-estimation
	- point-tracking
	- scene-flow
	- image-matching
	---

	# Kosmo-GFM: A Production-Ready Geometry Foundation Model

	Kosmo Research

	[Paper](https://arxiv.org/abs/2611.16840) · [Project](https://kosmoresearch.github.io/Kosmo-GFM/) · [Code](https://github.com/KosmoResearch/Kosmo-GFM) · [Checkpoints](https://huggingface.co/KosmoResearch/Kosmo-GFM/tree/main)

	![Kosmo-GFM: static scene reconstructions and dynamic 3D motion trajectories](assets/kosmo-gfm-overview.png)

	## Overview

	Kosmo-GFM predicts camera parameters, depth, cross-view correspondences, and
	dynamic 3D points from images and videos. A shared multi-view encoder supports
	task-specific geometry, motion, and matching decoders, with optional camera
	conditioning for reconstruction from known poses.

	The code repository provides standalone inference for GFM, Camera Condition GFM,
	GFM 4D, and MVLoMA. The full PoseEngine and Gaussian Splatting pipelines
	described in the technical report are not included.

	## Model Cards

	All geometry models use a ViT-g backbone. Parameter counts are rounded for the
	provided checkpoints. ✅ indicates a supported capability; — indicates it is
	not part of that inference configuration.

	\| Model \| Params \| Depth \| Camera Est. \| Camera Cond. \| 4D Motion \| Matching \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| [GFM](#gfm) \| 1.66B \| ✅ \| ✅ \| — \| — \| — \|
	\| [Camera Condition GFM](#camera-condition-gfm) \| 1.77B \| ✅ \| ✅ \| ✅ \| — \| — \|
	\| [GFM 4D](#gfm-4d) \| 1.65B \| ✅ \| ✅ \| — \| ✅ \| — \|
	\| [GFM + MVLoMA](#gfm-with-mvloma) \| 1.73B \| ✅ \| ✅ \| — \| — \| ✅ \|

	- Camera Condition GFM is the Camera-pose scale version and takes
	camera extrinsics and intrinsics as input.
	- GFM + MVLoMA shares the GFM encoder and adds a 75.9M-parameter matching
	head. It requires both the GFM and MVLoMA checkpoints.
	- GFM 4D predicts geometry and cross-frame motion for Dynamic Points and
	Scene Flow evaluation.

	\| Checkpoint \| Inference configuration \|
	\| --- \| --- \|
	\| [`gfm_3d.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/gfm_3d.pth) \| GFM: depth, cameras, and point clouds \|
	\| [`gfm_3d_camera_cond.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/gfm_3d_camera_cond.pth) \| Camera Condition GFM: Camera-pose scale version \|
	\| [`gfm_4d.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/gfm_4d.pth) \| GFM 4D: geometry and cross-frame motion \|
	\| [`mvloma.pth`](https://huggingface.co/KosmoResearch/Kosmo-GFM/blob/main/mvloma.pth) \| MVLoMA matching head; requires `gfm_3d.pth` \|

	## Quick Start

	Download the code and checkpoints, then run inference from the code repository root.

	### Installation

	Requires Python 3.10+ and a CUDA GPU. Install a compatible PyTorch / torchvision
	build, then:

	```bash
	git clone https://github.com/KosmoResearch/Kosmo-GFM.git
	cd Kosmo-GFM
	pip install -e .
	pip install -U huggingface_hub
	# Optional PLY/GLB export
	pip install -r requirements-optional.txt
	```

	### Download Checkpoints

	Download all four checkpoints into the `weights/` directory. The checkpoint
	manifest maps each inference configuration to its required weight files:

	```python
	import json
	from pathlib import Path
	from huggingface_hub import hf_hub_download, snapshot_download

	repo_id = "KosmoResearch/Kosmo-GFM"
	config_path = hf_hub_download(
	repo_id=repo_id, filename="config.json", local_dir="weights"
	)
	config = json.loads(Path(config_path).read_text())

	selected_models = list(config["models"]) # All inference configurations.
	# For GFM + MVLoMA only: selected_models = ["gfm_3d_with_mvloma"]
	checkpoints = sorted({
	filename
	for name in selected_models
	for filename in config["models"][name]["checkpoints"]
	})
	snapshot_download(
	repo_id=repo_id, local_dir="weights", allow_patterns=checkpoints
	)
	```

	To download a subset, set `selected_models` to any of `gfm_3d`,
	`gfm_3d_camera_cond`, `gfm_4d`, or `gfm_3d_with_mvloma`.
	The MVLoMA configuration includes both `gfm_3d.pth` and `mvloma.pth`.
	[`config.json`](config.json) is a checkpoint manifest for the standalone GFM
	scripts; it is not a Transformers `AutoModel` configuration.

	See the [Hugging Face download guide](https://huggingface.co/docs/huggingface_hub/en/guides/download)
	for authentication and download options.

	These are PyTorch checkpoints loaded by the GFM inference scripts.

	### Inference

	Run from the repository root with your own images or video frames.

	#### GFM

	```bash
	python tools/infer_3d.py \
	--checkpoint weights/gfm_3d.pth --input /path/to/images \
	--output outputs/gfm --save-npz --no-ply --no-glb
	```

	Exports depth, confidence, cameras and point clouds to `predictions.npz`.
	The default backbone is `vitg`; use `--model-size` to match your checkpoint.

	#### Camera Condition GFM

	The provided checkpoint is the Camera-pose scale version.

	```bash
	python tools/infer_3d_camera_cond.py \
	--checkpoint weights/gfm_3d_camera_cond.pth --input /path/to/images \
	--camera-npz /path/to/cameras.npz --output outputs/gfm_camera \
	--save-npz --no-ply --no-glb
	```

	The camera NPZ requires `extrinsics` (`[N,4,4]` or `[N,3,4]`, OpenCV
	world-to-camera) and `intrinsics` (`[N,3,3]`, at original image resolution).
	Cameras follow filename-sorted image order unless `image_names` is supplied.
	Use `--camera-convention c2w` for camera-to-world matrices.

	#### GFM 4D

	```bash
	python tools/infer_4d.py \
	--checkpoint weights/gfm_4d.pth --input /path/to/frames \
	--output outputs/gfm_4d --save-npz --no-ply --no-glb
	```

	Exports geometry, cameras and motion to `predictions.npz` at relative scene scale.
	Use `--chunk-size` to control query decoding memory.

	#### GFM with MVLoMA

	```bash
	python tools/infer_3d_with_mvloma.py \
	--checkpoint weights/gfm_3d.pth --mvloma-checkpoint weights/mvloma.pth \
	--input /path/to/images --output outputs/gfm_with_mvloma --no-ply --no-glb
	```

	Exports geometry to `predictions.npz` and image matches to `matches.npz`.
	Requires at least two images. Use `--all-pairs` to export every image pair.

	For PLY/GLB output, install the optional dependencies and omit
	`--no-ply --no-glb`. Each script provides additional options through `--help`.

	## Benchmarks

	### SpatialBench

	GFM results on SpatialBench.
	Higher is better for ↑; lower is better for ↓. Accuracy and AUC values are
	reported on a 0–1 scale.

	Depth

	\| Setting \| AbsRel↓ \| SqRel↓ \| RMSE↓ \| δ1.03↑ \| δ1.05↑ \| δ1.10↑ \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| Sparse \| 0.0630 \| 0.1093 \| 0.6207 \| 0.6696 \| 0.7810 \| 0.8816 \|
	\| Medium \| 0.0594 \| 0.0884 \| 0.5822 \| 0.6895 \| 0.8077 \| 0.9025 \|

	Camera

	\| Setting \| RAcc3↑ \| RAcc5↑ \| TAcc3↑ \| TAcc5↑ \| AUC@5↑ \| AUC@15↑ \| AUC@30↑ \|
	\| --- \| --- \| --- \| --- \| --- \| --- \| --- \| --- \|
	\| Sparse \| 0.8358 \| 0.9055 \| 0.6457 \| 0.7394 \| 0.5726 \| 0.7412 \| 0.8226 \|
	\| Medium \| 0.8310 \| 0.8938 \| 0.6468 \| 0.7322 \| 0.5820 \| 0.7450 \| 0.8281 \|

	See [SpatialBench](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/bench/SpatialBench/README.md) for evaluation usage.

	### GFM4DBench

	GFM results on GFM4DBench.

	\| Dataset \| Point APD (%)↑ \| Point EPE (m)↓ \| Flow Accuracy (%)↑ \| Flow EPE (m)↓ \|
	\| --- \| --- \| --- \| --- \| --- \|
	\| Panoptic Studio \| 83.41 \| 0.1411 \| 50.15 \| 0.1257 \|
	\| PointOdyssey \| 85.73 \| 0.1474 \| 83.02 \| 0.0521 \|
	\| Dynamic Replica \| 80.76 \| 0.2017 \| 91.31 \| 0.0279 \|
	\| Aria Digital Twin \| 89.78 \| 0.1102 \| 96.34 \| 0.0134 \|
	\| SynthVerse \| 42.49 \| 1.1105 \| 34.56 \| 0.7213 \|
	\| Mean \| 76.43 \| 0.3422 \| 71.08 \| 0.1881 \|

	Mean is the arithmetic average across all five datasets.

	See [GFM4DBench](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/bench/GFM4DBench/README.md) for Dynamic Points and Scene Flow
	evaluation on TapVid3D / WorldTrack and SynthVerse.

	## Acknowledgements

	Thanks to [VGGT](https://github.com/facebookresearch/vggt),
	[DINOv2](https://github.com/facebookresearch/dinov2),
	[Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3),
	[DPT](https://github.com/isl-org/DPT), [DeDoDe](https://github.com/Parskatt/DeDoDe),
	[DaD](https://github.com/Parskatt/DaD), [LightGlue](https://github.com/cvg/LightGlue),
	[SpatialBench](https://github.com/Ropedia/SpatialBench), and
	[Open-d4rt](https://github.com/Lijiaxin0111/Open-d4rt).

	## Citation

	```bibtex
	@techreport{kosmo2026gfm,
	title={{Kosmo-GFM}: A Production-Ready Geometry Foundation Model},
	author={{Kosmo Research}},
	institution={Kosmo Research},
	year={2026},
	url={https://arxiv.org/abs/2611.16840}
	}
	```

	## License

	Original code is licensed under [Apache-2.0](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/LICENSE). Third-party terms,
	including the [VGGT use restrictions](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/licenses/VGGT-LICENSE.txt), also apply;
	see [NOTICE](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/NOTICE) and [provenance notes](https://github.com/KosmoResearch/Kosmo-GFM/blob/main/licenses/README.md).