| --- |
| license: other |
| license_name: dinov3-license |
| license_link: https://ai.meta.com/resources/models-and-libraries/dinov3-license |
| base_model: facebook/dinov3-vitl16-pretrain-lvd1689m |
| base_model_relation: finetune |
| library_name: pytorch |
| tags: |
| - geolocation |
| - image-regression |
| - flow-matching |
| - dinov3 |
| - street-view |
| - jakarta |
| - computer-vision |
| language: |
| - id |
| datasets: |
| - nadh0708/JKTSV-primary |
| --- |
| |
| # Img2LocJakarta 2.0 — DINOv3 + flow matching street-view geolocation |
|
|
| Predict a **distribution** of **where a street-view image was taken in Jakarta**. A fine-tuned |
| **DINOv3 ViT-L/16** backbone encodes the image, then a conditional flow-matching (CFM) head |
| propagates Gaussian-noise particles onto the geographic coordinates. |
|
|
| | | | |
| |---|---| |
| | **Input** | RGB street-level image (any size, resized to 256×256) | |
| | **Output** | `num_particles` predicted coordinates (100 particles → 100 `lat/lon` points), as a list or GeoJSON | |
| | **Scope** | Jakarta street imagery only [primary road] | |
| | **Backbone** | DINOv3 ViT-L/16, fine-tuned end to end | |
| | **Head** | Velocity MLP (4 AdaLN residual blocks, hidden 512), Heun sampler | |
|
|
| ## Quick Start |
|
|
| ```bash |
| pip install torch torchvision pillow numpy huggingface_hub safetensors |
| ``` |
|
|
| `git` is needed on first run: the DINOv3 *architecture* source is cloned from |
| `facebookresearch/dinov3` into the torch hub cache (no extra weights are downloaded). |
|
|
|  |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| # Download the inference code and the example image |
| for fname in ["inference.py", "modeling_geotag.py", "examples/kokas.png"]: |
| hf_hub_download("nadh0708/Img2LocJakarta2.0", fname, local_dir=".") |
| |
| from inference import GeoTagPredictor |
| |
| predictor = GeoTagPredictor("nadh0708/Img2LocJakarta2.0") |
| coords = predictor.predict("examples/kokas.png", num_particles=100) |
| print(len(coords), coords[0]) |
| # 100 {'lat': -6.2257, 'lon': 106.8309} <- 100 predicted coordinates |
| |
| # Batched: one list of coordinates per image |
| clouds = predictor.predict(["a.jpg", "b.jpg"], num_particles=100) |
| |
| # Optional: mean point + uncertainty of one cloud |
| print(predictor.summarize(coords)) |
| # {'lat': -6.2246, 'lon': 106.8335, 'n_particles': 100, 'sigma_m': 484.3, |
| # 'p50_radius_m': 372.1, 'p90_radius_m': 734.0} |
| |
| # GeoJSON, one Point feature per coordinate |
| geojson = predictor.to_geojson(["examples/kokas.png"], num_particles=100) |
| ``` |
|
|
| The printed values are from a CUDA GPU with the default `seed=0`. On CPU the random draws |
| differ, so the cloud is slightly different (mean `-6.2247, 106.8339`, `sigma_m` 496.2). |
|
|
| ### From a local folder |
|
|
| ```python |
| predictor = GeoTagPredictor(".") # folder containing model.safetensors |
| ``` |
|
|
| ### CLI (GeoJSON output) |
|
|
| ```bash |
| python inference.py nadh0708/Img2LocJakarta2.0 examples/kokas.png --particles 100 -o out.geojson |
| python inference.py nadh0708/Img2LocJakarta2.0 a.jpg b.jpg --particles 512 --seed 1 |
| python inference.py nadh0708/Img2LocJakarta2.0 a.jpg --truth 106.83 -6.22 # adds error_m |
| python inference.py nadh0708/Img2LocJakarta2.0 a.jpg --table # text summary |
| ``` |
|
|
| `--particles N` gives N coordinates per image, so N Point features. Open `out.geojson` in |
| QGIS or geojson.io to see them on a map. |
|
|
| ### Low-level API |
|
|
| ```python |
| import torch |
| from PIL import Image |
| from torchvision.transforms import v2 |
| from modeling_geotag import DinoV3CFMRegressor |
| |
| model = DinoV3CFMRegressor.from_pretrained("nadh0708/Img2LocJakarta2.0") |
| |
| tf = v2.Compose([ |
| v2.ToImage(), |
| v2.Resize((256, 256), antialias=True), |
| v2.ToDtype(torch.float32, scale=True), |
| v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), |
| ]) |
| x = tf(Image.open("examples/kokas.png").convert("RGB")).unsqueeze(0) |
| cloud = model.sample_lonlat(x, num_particles=256, steps=8) # (1, 256, 2) [lon, lat] |
| ``` |
|
|
| ## Output |
|
|
| `predict(image, num_particles=N)` returns **N predicted coordinates (the predictive distribution)** |
| for the image, each `{"lat": ..., "lon": ...}` in WGS-84 degrees. |
|
|
| GeoJSON (`to_geojson` / CLI) is a `FeatureCollection` with one `Point` per coordinate |
| (properties: `image`, `particle`, plus `error_m` if a true location is given) and a |
| top-level `summary` list with one entry per image: |
|
|
| | `summary` field | Meaning | |
| |---|---| |
| | `lat`, `lon` | mean of the coordinates | |
| | `n_particles` | number of coordinates | |
| | `sigma_m` | RMS spread around the mean, metres | |
| | `p50_radius_m`, `p90_radius_m` | radius around the mean containing 50 % / 90 % of the coordinates | |
| | `error_m` | distance of the mean to `--truth`, when given | |
|
|
| `num_particles` *is* the number of samples drawn from the predicted coordinate distribution; |
| more particles give a smoother estimate. `seed` makes the cloud reproducible. |
|
|
| ## Performance |
|
|
| Evaluated on the **full test split of `JKTSV-primary`: 137,173 images** (8 headings per |
| location, 0°–315° in 45° steps), with this checkpoint (stage D, epoch 80). Setup: |
| 32 particles, Heun integrator. Error is the great-circle (haversine) distance between the |
| true location and the **mean of the particle cloud**. |
|
|
| | Metric | Value | |
| |---|---| |
| | Mean error | 1.05 km | |
| | Median error | 0.40 km | |
| | 75th percentile | 0.82 km | |
| | 90th percentile | 2.34 km | |
| | RMSE | 2.36 km | |
|
|
| | Within | 100 m | 250 m | 500 m | 1 km | 2 km | 5 km | |
| |---|---|---|---|---|---|---| |
| | % of test images | 6.9 | 30.4 | 59.3 | 79.4 | 88.7 | 95.2 | |
|
|
| ### How the error shrinks while the flow runs |
|
|
| Mean particle error over the test set: |
|
|
| | Flow time t | 0.0 | 0.2 | 0.4 | 0.5 | 0.6 | 0.8 | 1.0 | |
| |---|---|---|---|---|---|---|---| |
| | Mean error (km) | 11.7 | 8.9 | 3.9 | 2.4 | 1.7 | 1.4 | 1.4 | |
|
|
| #### Note |
| * Individual particles are noisier than their mean, so the cloud mean is the better single |
| point estimate. |
| * Numbers were measured with 32 particles; the default in `predict` is 256. |
|
|
| ## Architecture |
|
|
| ``` |
| DinoV3CFMRegressor |
| ├─ encoder_net : DINOv3 ViT-L/16 (fine-tuned), CLS token -> (B, 1024) |
| ├─ velocity_head : v(y_t, t, cond) -> R^2 |
| │ [cond ‖ y_t] -> 512 -> 4 × AdaLN residual block (time-conditioned) -> 2 |
| └─ target_mean/var: standardiser of the local (x, y) metre targets (stored in the weights) |
| |
| sampling: y_0 ~ N(0, I) --Heun, 8 steps--> y_1 --de-standardise--> (x, y) metres |
| --inverse flat-earth projection--> (lon, lat) |
| ``` |
|
|
| The meter offsets are relative to the Jakarta origin `(lon 106.828320, lat -6.227468)`. |
| The checkpoint bundles the fine-tuned backbone and the head, so no separate LVD-1689M |
| weight download is required. |
|
|
| ## Training |
|
|
| Final checkpoint: stage D (full-backbone fine-tuning), epoch 80. Best validation RMSE |
| logged during training: 2.32 km. Trained on |
| [`nadh0708/JKTSV-primary`](https://huggingface.co/datasets/nadh0708/JKTSV-primary) |
| (Google Street View along Jakarta motorway and primary roads). The backbone starts from |
| Meta's [DINOv3 ViT-L/16 (LVD-1689M)](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m). |
|
|
| ## Limitations |
|
|
| - **Geographically restricted to Jakarta.** |
| - Uncertainty is not calibrated against a likelihood. |
| - Trained on street-view cutouts; other camera settings may degrade accuracy. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `modeling_geotag.py` | Model classes, Heun sampler, `from_pretrained` loader, coordinate conversion | |
| | `inference.py` | `GeoTagPredictor` high-level API + GeoJSON CLI | |
| | `model.safetensors` | Full state dict (fine-tuned backbone + head + standardiser, ~1.2 GB) | |
| | `requirements.txt` | Runtime dependencies | |
| | `examples/kokas.png` | Example street-view image used in the Quick Start | |
| | `LICENSE` | Apache License 2.0 (the code) | |
| | `LICENSE-DINOv3.md` | DINOv3 License (the weights) | |
|
|
| ## Acknowledgment |
|
|
| Thanks to Meta for this onderful model & Claude for helping me write the inference code for this repo. |
|
|
| ## Citation |
|
|
| The backbone is DINOv3: |
|
|
| ```bibtex |
| @misc{simeoni2025dinov3, |
| title={{DINOv3}}, |
| author={Sim{\'e}oni, Oriane and Vo, Huy V. and Seitzer, Maximilian and Baldassarre, Federico and Oquab, Maxime and Jose, Cijo and Khalidov, Vasil and Szafraniec, Marc and Yi, Seungeun and Ramamonjisoa, Micha{\"e}l and Massa, Francisco and Haziza, Daniel and Wehrstedt, Luca and Wang, Jianyuan and Darcet, Timoth{\'e}e and Moutakanni, Th{\'e}o and Sentana, Leonel and Roberts, Claire and Vedaldi, Andrea and Tolan, Jamie and Brandt, John and Couprie, Camille and Mairal, Julien and J{\'e}gou, Herv{\'e} and Labatut, Patrick and Bojanowski, Piotr}, |
| year={2025}, |
| eprint={2508.10104}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2508.10104}, |
| } |
| ``` |
|
|
| ## License |
|
|
| - **Code** (`inference.py`, `modeling_geotag.py`): Apache License 2.0, see [`LICENSE`](LICENSE). |
| - **Weights** (`model.safetensors`): the backbone is fine-tuned from Meta's DINOv3, so the |
| weights are a derivative of the DINO Materials and are distributed under the |
| **DINOv3 License**, see [`LICENSE-DINOv3.md`](LICENSE-DINOv3.md). |
| - The DINOv3 architecture source is fetched from `facebookresearch/dinov3` at runtime and |
| remains under its own license. |
|
|
