README.md

metadata

language:
  - en
license: apache-2.0
library_name: pytorch
pipeline_tag: depth-estimation
tags:
  - computer-vision
  - multi-view-reconstruction
  - camera-pose-estimation
  - point-tracking
  - scene-flow
  - image-matching

Kosmo-GFM: A Production-Ready Geometry Foundation Model

Kosmo Research

Paper · Project · Code · Checkpoints

Overview

Kosmo-GFM predicts camera parameters, depth, cross-view correspondences, and dynamic 3D points from images and videos. A shared multi-view encoder supports task-specific geometry, motion, and matching decoders, with optional camera conditioning for reconstruction from known poses.

The code repository provides standalone inference for GFM, Camera Condition GFM, GFM 4D, and MVLoMA. The full PoseEngine and Gaussian Splatting pipelines described in the technical report are not included.

Model Cards

All geometry models use a ViT-g backbone. Parameter counts are rounded for the provided checkpoints. ✅ indicates a supported capability; — indicates it is not part of that inference configuration.

Model	Params	Depth	Camera Est.	Camera Cond.	4D Motion	Matching
GFM	1.66B	✅	✅	—	—	—
Camera Condition GFM	1.77B	✅	✅	✅	—	—
GFM 4D	1.65B	✅	✅	—	✅	—
GFM + MVLoMA	1.73B	✅	✅	—	—	✅

• Camera Condition GFM is the Camera-pose scale version and takes camera extrinsics and intrinsics as input.
• GFM + MVLoMA shares the GFM encoder and adds a 75.9M-parameter matching head. It requires both the GFM and MVLoMA checkpoints.
• GFM 4D predicts geometry and cross-frame motion for Dynamic Points and Scene Flow evaluation.

Checkpoint	Inference configuration
gfm_3d.pth	GFM: depth, cameras, and point clouds
gfm_3d_camera_cond.pth	Camera Condition GFM: Camera-pose scale version
gfm_4d.pth	GFM 4D: geometry and cross-frame motion
mvloma.pth	MVLoMA matching head; requires gfm_3d.pth

Quick Start

Download the code and checkpoints, then run inference from the code repository root.

Installation

Requires Python 3.10+ and a CUDA GPU. Install a compatible PyTorch / torchvision build, then:

git clone https://github.com/KosmoResearch/Kosmo-GFM.git
cd Kosmo-GFM
pip install -e .
pip install -U huggingface_hub
# Optional PLY/GLB export
pip install -r requirements-optional.txt

Download Checkpoints

Download all four checkpoints into the weights/ directory:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="KosmoResearch/Kosmo-GFM",
    local_dir="weights",
    allow_patterns=[
        "gfm_3d.pth",
        "gfm_3d_camera_cond.pth",
        "gfm_4d.pth",
        "mvloma.pth",
    ],
)

To download only the checkpoints you need, reduce allow_patterns accordingly. See the Hugging Face download guide for authentication and download options.

These are PyTorch checkpoints loaded by the GFM inference scripts.

Inference

Run from the repository root with your own images or video frames.

GFM

python tools/infer_3d.py \
  --checkpoint weights/gfm_3d.pth --input /path/to/images \
  --output outputs/gfm --save-npz --no-ply --no-glb

Exports depth, confidence, cameras and point clouds to predictions.npz. The default backbone is vitg; use --model-size to match your checkpoint.

Camera Condition GFM

The provided checkpoint is the Camera-pose scale version.

python tools/infer_3d_camera_cond.py \
  --checkpoint weights/gfm_3d_camera_cond.pth --input /path/to/images \
  --camera-npz /path/to/cameras.npz --output outputs/gfm_camera \
  --save-npz --no-ply --no-glb

The camera NPZ requires extrinsics ([N,4,4] or [N,3,4], OpenCV world-to-camera) and intrinsics ([N,3,3], at original image resolution). Cameras follow filename-sorted image order unless image_names is supplied. Use --camera-convention c2w for camera-to-world matrices.

GFM 4D

python tools/infer_4d.py \
  --checkpoint weights/gfm_4d.pth --input /path/to/frames \
  --output outputs/gfm_4d --save-npz --no-ply --no-glb

Exports geometry, cameras and motion to predictions.npz at relative scene scale. Use --chunk-size to control query decoding memory.

GFM with MVLoMA

python tools/infer_3d_with_mvloma.py \
  --checkpoint weights/gfm_3d.pth --mvloma-checkpoint weights/mvloma.pth \
  --input /path/to/images --output outputs/gfm_with_mvloma --no-ply --no-glb

Exports geometry to predictions.npz and image matches to matches.npz. Requires at least two images. Use --all-pairs to export every image pair.

For PLY/GLB output, install the optional dependencies and omit --no-ply --no-glb. Each script provides additional options through --help.

Benchmarks

SpatialBench

GFM results on SpatialBench. Higher is better for ↑; lower is better for ↓. Accuracy and AUC values are reported on a 0–1 scale.

Depth

Setting	AbsRel↓	SqRel↓	RMSE↓	δ1.03↑	δ1.05↑	δ1.10↑
Sparse	0.0630	0.1093	0.6207	0.6696	0.7810	0.8816
Medium	0.0594	0.0884	0.5822	0.6895	0.8077	0.9025

Camera

Setting	RAcc3↑	RAcc5↑	TAcc3↑	TAcc5↑	AUC@5↑	AUC@15↑	AUC@30↑
Sparse	0.8358	0.9055	0.6457	0.7394	0.5726	0.7412	0.8226
Medium	0.8310	0.8938	0.6468	0.7322	0.5820	0.7450	0.8281

See SpatialBench for evaluation usage.

GFM4DBench

GFM results on GFM4DBench.

Dataset	Point APD (%)↑	Point EPE (m)↓	Flow Accuracy (%)↑	Flow EPE (m)↓
Panoptic Studio	83.41	0.1411	50.15	0.1257
PointOdyssey	85.73	0.1474	83.02	0.0521
Dynamic Replica	80.76	0.2017	91.31	0.0279
Aria Digital Twin	89.78	0.1102	96.34	0.0134
SynthVerse	42.49	1.1105	34.56	0.7213
Mean	76.43	0.3422	71.08	0.1881

Mean is the arithmetic average across all five datasets.

See GFM4DBench for Dynamic Points and Scene Flow evaluation on TapVid3D / WorldTrack and SynthVerse.

Acknowledgements

Thanks to VGGT, DINOv2, Depth Anything 3, DPT, DeDoDe, DaD, LightGlue, SpatialBench, and Open-d4rt.

Citation

@techreport{kosmo2026gfm,
  title={{Kosmo-GFM}: A Production-Ready Geometry Foundation Model},
  author={{Kosmo Research}},
  institution={Kosmo Research},
  year={2026},
  url={https://arxiv.org/abs/2611.16840}
}

License

Original code is licensed under Apache-2.0. Third-party terms, including the VGGT use restrictions, also apply; see NOTICE and provenance notes.