Blame for README.md

---
license: other
license_name: dinov3-license
license_link: https://ai.meta.com/resources/models-and-libraries/dinov3-license
base_model: facebook/dinov3-vitl16-pretrain-lvd1689m
base_model_relation: finetune
library_name: pytorch
tags:
  - geolocation
  - image-regression
  - flow-matching
  - dinov3
  - street-view
  - jakarta
  - computer-vision
language:
  - id
datasets:
  - nadh0708/JKTSV-primary
---

# Img2LocJakarta 2.0 — DINOv3 + flow matching street-view geolocation

Predict a **distribution** of **where a street-view image was taken in Jakarta**. A fine-tuned
**DINOv3 ViT-L/16** backbone encodes the image, then a conditional flow-matching (CFM) head
propagates Gaussian-noise particles onto the geographic coordinates.

| | |
|---|---|
| **Input** | RGB street-level image (any size, resized to 256×256) |
| **Output** | `num_particles` predicted coordinates (100 particles → 100 `lat/lon` points), as a list or GeoJSON |
| **Scope** | Jakarta street imagery only [primary road] |
| **Backbone** | DINOv3 ViT-L/16, fine-tuned end to end |
| **Head** | Velocity MLP (4 AdaLN residual blocks, hidden 512), Heun sampler |

## Quick Start

```bash
pip install torch torchvision pillow numpy huggingface_hub safetensors
```

`git` is needed on first run: the DINOv3 *architecture* source is cloned from
`facebookresearch/dinov3` into the torch hub cache (no extra weights are downloaded).

![Example street-view image, examples/kokas.png](examples/kokas.png)

```python
from huggingface_hub import hf_hub_download

# Download the inference code and the example image
for fname in ["inference.py", "modeling_geotag.py", "examples/kokas.png"]:
    hf_hub_download("nadh0708/Img2LocJakarta2.0", fname, local_dir=".")

from inference import GeoTagPredictor

predictor = GeoTagPredictor("nadh0708/Img2LocJakarta2.0")
coords = predictor.predict("examples/kokas.png", num_particles=100)
print(len(coords), coords[0])
# 100 {'lat': -6.2257, 'lon': 106.8309}      <- 100 predicted coordinates

# Batched: one list of coordinates per image
clouds = predictor.predict(["a.jpg", "b.jpg"], num_particles=100)

# Optional: mean point + uncertainty of one cloud
print(predictor.summarize(coords))
# {'lat': -6.2246, 'lon': 106.8335, 'n_particles': 100, 'sigma_m': 484.3,
#  'p50_radius_m': 372.1, 'p90_radius_m': 734.0}

# GeoJSON, one Point feature per coordinate
geojson = predictor.to_geojson(["examples/kokas.png"], num_particles=100)
```

The printed values are from a CUDA GPU with the default `seed=0`. On CPU the random draws
differ, so the cloud is slightly different (mean `-6.2247, 106.8339`, `sigma_m` 496.2).

### From a local folder

```python
predictor = GeoTagPredictor(".")          # folder containing model.safetensors
```

### CLI (GeoJSON output)

```bash
python inference.py nadh0708/Img2LocJakarta2.0 examples/kokas.png --particles 100 -o out.geojson
python inference.py nadh0708/Img2LocJakarta2.0 a.jpg b.jpg --particles 512 --seed 1
python inference.py nadh0708/Img2LocJakarta2.0 a.jpg --truth 106.83 -6.22   # adds error_m
python inference.py nadh0708/Img2LocJakarta2.0 a.jpg --table                # text summary
```

`--particles N` gives N coordinates per image, so N Point features. Open `out.geojson` in
QGIS or geojson.io to see them on a map.

### Low-level API

```python
import torch
from PIL import Image
from torchvision.transforms import v2
from modeling_geotag import DinoV3CFMRegressor

model = DinoV3CFMRegressor.from_pretrained("nadh0708/Img2LocJakarta2.0")

tf = v2.Compose([
    v2.ToImage(),
    v2.Resize((256, 256), antialias=True),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])
x = tf(Image.open("examples/kokas.png").convert("RGB")).unsqueeze(0)
cloud = model.sample_lonlat(x, num_particles=256, steps=8)   # (1, 256, 2) [lon, lat]
```

## Output

`predict(image, num_particles=N)` returns **N predicted coordinates (the predictive distribution)**
for the image, each `{"lat": ..., "lon": ...}` in WGS-84 degrees.

GeoJSON (`to_geojson` / CLI) is a `FeatureCollection` with one `Point` per coordinate
(properties: `image`, `particle`, plus `error_m` if a true location is given) and a
top-level `summary` list with one entry per image:

| `summary` field | Meaning |
|---|---|
| `lat`, `lon` | mean of the coordinates |
| `n_particles` | number of coordinates |
| `sigma_m` | RMS spread around the mean, metres |
| `p50_radius_m`, `p90_radius_m` | radius around the mean containing 50 % / 90 % of the coordinates |
| `error_m` | distance of the mean to `--truth`, when given |

`num_particles` *is* the number of samples drawn from the predicted coordinate distribution;
more particles give a smoother estimate. `seed` makes the cloud reproducible.

## Performance

Evaluated on the **full test split of `JKTSV-primary`: 137,173 images** (8 headings per
location, 0°–315° in 45° steps), with this checkpoint (stage D, epoch 80). Setup:
32 particles, Heun integrator. Error is the great-circle (haversine) distance between the
true location and the **mean of the particle cloud**.

| Metric | Value |
|---|---|
| Mean error | 1.05 km |
| Median error | 0.40 km |
| 75th percentile | 0.82 km |
| 90th percentile | 2.34 km |
| RMSE | 2.36 km |

| Within | 100 m | 250 m | 500 m | 1 km | 2 km | 5 km |
|---|---|---|---|---|---|---|
| % of test images | 6.9 | 30.4 | 59.3 | 79.4 | 88.7 | 95.2 |

### How the error shrinks while the flow runs

Mean particle error over the test set:

| Flow time t | 0.0 | 0.2 | 0.4 | 0.5 | 0.6 | 0.8 | 1.0 |
|---|---|---|---|---|---|---|---|
| Mean error (km) | 11.7 | 8.9 | 3.9 | 2.4 | 1.7 | 1.4 | 1.4 |

#### Note
* Individual particles are noisier than their mean, so the cloud mean is the better single
  point estimate.
* Numbers were measured with 32 particles; the default in `predict` is 256.

## Architecture

```
DinoV3CFMRegressor
 ├─ encoder_net    : DINOv3 ViT-L/16 (fine-tuned), CLS token -> (B, 1024)
 ├─ velocity_head  : v(y_t, t, cond) -> R^2
 │                    [cond ‖ y_t] -> 512 -> 4 × AdaLN residual block (time-conditioned) -> 2
 └─ target_mean/var: standardiser of the local (x, y) metre targets (stored in the weights)

sampling: y_0 ~ N(0, I)  --Heun, 8 steps-->  y_1  --de-standardise-->  (x, y) metres
          --inverse flat-earth projection-->  (lon, lat)
```

The meter offsets are relative to the Jakarta origin `(lon 106.828320, lat -6.227468)`.
The checkpoint bundles the fine-tuned backbone and the head, so no separate LVD-1689M
weight download is required.

## Training

Final checkpoint: stage D (full-backbone fine-tuning), epoch 80. Best validation RMSE
logged during training: 2.32 km. Trained on
[`nadh0708/JKTSV-primary`](https://huggingface.co/datasets/nadh0708/JKTSV-primary)
(Google Street View along Jakarta motorway and primary roads). The backbone starts from
Meta's [DINOv3 ViT-L/16 (LVD-1689M)](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m).

## Limitations

- **Geographically restricted to Jakarta.**
- Uncertainty is not calibrated against a likelihood.
- Trained on street-view cutouts; other camera settings may degrade accuracy.

## Files

| File | Purpose |
|---|---|
| `modeling_geotag.py` | Model classes, Heun sampler, `from_pretrained` loader, coordinate conversion |
| `inference.py` | `GeoTagPredictor` high-level API + GeoJSON CLI |
| `model.safetensors` | Full state dict (fine-tuned backbone + head + standardiser, ~1.2 GB) |
| `requirements.txt` | Runtime dependencies |
| `examples/kokas.png` | Example street-view image used in the Quick Start |
| `LICENSE` | Apache License 2.0 (the code) |
| `LICENSE-DINOv3.md` | DINOv3 License (the weights) |

## Acknowledgment

Thanks to Meta for this onderful model & Claude for helping me write the inference code for this repo.

## Citation

The backbone is DINOv3:

```bibtex
@misc{simeoni2025dinov3,
  title={{DINOv3}},
  author={Sim{\'e}oni, Oriane and Vo, Huy V. and Seitzer, Maximilian and Baldassarre, Federico and Oquab, Maxime and Jose, Cijo and Khalidov, Vasil and Szafraniec, Marc and Yi, Seungeun and Ramamonjisoa, Micha{\"e}l and Massa, Francisco and Haziza, Daniel and Wehrstedt, Luca and Wang, Jianyuan and Darcet, Timoth{\'e}e and Moutakanni, Th{\'e}o and Sentana, Leonel and Roberts, Claire and Vedaldi, Andrea and Tolan, Jamie and Brandt, John and Couprie, Camille and Mairal, Julien and J{\'e}gou, Herv{\'e} and Labatut, Patrick and Bojanowski, Piotr},
  year={2025},
  eprint={2508.10104},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2508.10104},
}
```

## License

- **Code** (`inference.py`, `modeling_geotag.py`): Apache License 2.0, see [`LICENSE`](LICENSE).
- **Weights** (`model.safetensors`): the backbone is fine-tuned from Meta's DINOv3, so the
  weights are a derivative of the DINO Materials and are distributed under the
  **DINOv3 License**, see [`LICENSE-DINOv3.md`](LICENSE-DINOv3.md).
- The DINOv3 architecture source is fetched from `facebookresearch/dinov3` at runtime and
  remains under its own license.