litecalib-vitb / README.md
kocurvik's picture
Card: CommonsPano as one row, its by / by-sa subsets
263cbb1 verified
|
Raw History Blame Contribute Delete
4.76 kB
---
license: apache-2.0
library_name: litecalib
tags:
- camera-calibration
- single-view-calibration
- computer-vision
---
# litecalib_vitb
A released model of **LiteCalib: Fast, Model-Agnostic Single-View Camera Calibration** (Viktor Kocur, Lukáš Bujňák, Lukáš Gajdošech, Torsten Sattler, Zuzana Kukelova).
LiteCalib with a DINOv2 ViT-B/14 backbone: the camera's field of view at a 5 x 5 grid of image points, to which any radially symmetric camera model is fitted. 106 M parameters.
Code, documentation and training: [github.com/kocurvik/litecalib](https://github.com/kocurvik/litecalib).
```bash
pip install litecalib
```
```python
import litecalib
model = litecalib.load("litecalib_vitb")
calib = model.calibrate("photo.jpg", "radial:1")
print(calib.K, calib.distortion)
```
## Accuracy and runtime
Averaged over the ten benchmarks of the paper (Stanford2D3D, TartanAir, MegaDepth, LaMAR, MegaDepth-radial, ScanNet++, Mono, Cathedral, InFlux and InFlux++), each image fitted with its benchmark's camera model:
| median angular error | angular error mAA@5° | median vFoV error | vFoV error mAA@5° |
|---|---|---|---|
| 1.50° | 66.6 % | 3.20° | 43.7 % |
The angular error is AnyCalib's: the angle between the predicted and the true ray, over the pixels of the image;
mAA@5° is the area under its recall curve up to 5°.
Runtime per image, batch 1 (network + camera fit):
| device | network | fit | total |
|---|---|---|---|
| NVIDIA A100, PyTorch fp16 | 6.3 ms | 0.3 ms (CPU) | 6.6 ms |
| Jetson Orin Nano 8 GB (15 W), TensorRT fp16 | 22.9 ms | 4.2 ms (CPU) | 27.1 ms |
The fit is the mean over the camera models; the Jetson's network is a TensorRT engine of the exported model,
which this package does not build.
## Training data
Trained from DINOv2's weights for 24 epochs (warmup-stable-decay schedule), each epoch 144,000 views: 72,000 drawn
from the panorama views and 72,000 from MegaScenes. All the released models were trained on the same data.
| source | images | views (train / val) | cameras | licences |
|---|---|---|---|---|
| OpenPano, AnyCalib's extended version | 3,651 / 202 panoramas | 51,719 / 2,872 | pinhole, Brown-Conrady (`radial:1`), EUCM, about a third each | per source (below) |
| CommonsPano (`by` and `by-sa` subsets) | 9,336 / 555 panoramas | 148,626 / 8,851 | as OpenPano | public domain, CC0, CC BY (`by`); CC BY-SA (`by-sa`) |
| MegaScenes | 14,382 / 160 scenes | 1,211,520 / 10,779 | `radial:1` from COLMAP's `SIMPLE_RADIAL` | per photo, from Wikimedia Commons (74 % CC BY-SA, 17 % CC BY, 8 % public domain / CC0) |
- **OpenPano**: introduced by GeoCalib, which describes how its panoramas were collected and split
([GeoCalib: OpenPano Dataset](https://github.com/cvg/GeoCalib?tab=readme-ov-file#openpano-dataset)); extended by
AnyCalib with panoramas that need not be aligned with gravity
([AnyCalib: Extended OpenPano Dataset](https://github.com/javrtg/AnyCalib?tab=readme-ov-file#extended-openpano-dataset)).
It consists of tonemapped panoramas from [The Laval Photometric Indoor HDR Dataset](http://hdrdb.com/indoor-hdr-photometric/),
[PolyHaven](https://polyhaven.com/hdris), [HDRMaps](https://hdrmaps.com/freebies/free-hdris/),
[AmbientCG](https://ambientcg.com/list?type=hdri&sort=popular) and
[BlenderKit](https://www.blenderkit.com/asset-gallery?query=category_subtree:hdr). AnyCalib's splits.
- **CommonsPano** ([LiteCalib/CommonsPano](https://huggingface.co/datasets/LiteCalib/CommonsPano)): 360°
panoramas from Wikimedia Commons, split by uploader, with a per-file attribution list. 465 duplicates were
removed: 444 of OpenPano panoramas (found by image content and by source name) and 21 within Commons. Three
training panoramas yield no valid view, so the views come from 9,333 of them.
- **Panorama views**: 640 x 640 perspective and distorted views rendered from each panorama with AnyCalib's camera
sampling (its `OP_g` recipe), principal point at the center, square pixels.
- **MegaScenes** ([megascenes.github.io](https://megascenes.github.io)): Wikimedia Commons photos with COLMAP
reconstructions. Each photo is center-cropped to 1:1, 4:3 or 3:2 at a short side of 600 px, its principal point
at the center. Left out: reconstructions with fewer than 20 photos, failed self-calibrations, and the scenes of the
MegaDepth evaluation landmarks.
- **Training**: AnyCalib's resolution (about 102,400 pixels, aspect ratio 0.5 to 2) and its photometric
augmentations.
More in the [repository](https://github.com/kocurvik/litecalib#datasets).
## License
Apache License 2.0. We thank the authors of [The Laval Photometric Indoor HDR Dataset](http://hdrdb.com/indoor-hdr-photometric/) for allowing us to
release these weights under this license.