| --- |
| license: apache-2.0 |
| pipeline_tag: image-feature-extraction |
| --- |
| |
| # GAIR: Location-Aware Self-Supervised Contrastive Pre-Training with Geo-Aligned Implicit Representations |
|
|
| Official implementation of GAIR, a location-aware self-supervised pre-training framework for learning geo-aligned representations from remote sensing imagery, street-view imagery, and geographic coordinates. |
|
|
| [**Paper**](https://huggingface.co/papers/2503.16683) | [**GitHub**](https://github.com/zpl99/GAIR) |
|
|
| ## Overview |
| GAIR solves the representation problem of detailed localized image representations at arbitrary positions by using an implicit neural representation (INR) module extending ViT with Neural Implicit Local Interpolation (NILI). It integrates overhead remote sensing (RS) data, street view (SV) imagery, and their geolocation metadata for contrastive learning. The model produces continuous RS image representations covering arbitrary locations. |
|
|
| ## Sample Usage |
|
|
| To use this model, ensure you have the dependencies installed and the `gair` module from the official repository in your path. |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from gair import GAIRModel, load_rs_image, load_sv_image, preprocess_rs_array, preprocess_sv_array |
| |
| # Download the checkpoint |
| checkpoint = hf_hub_download(repo_id="PingL/GAIR", filename="checkpoint.pth") |
| device = "cuda:0" if torch.cuda.is_available() else "cpu" |
| |
| # Load the model |
| model = GAIRModel.from_checkpoint(checkpoint, device=device, query_mode="nili") |
| |
| # Prepare inputs |
| rs = preprocess_rs_array(load_rs_image("path/to/rs_crop.tif")).to(device) |
| sv = preprocess_sv_array(load_sv_image("path/to/street_view.jpg")).to(device) |
| coords = torch.tensor([[116.397, 39.908]], device=device) # [lon, lat] |
| bbox = torch.tensor([[116.387, 39.918, 116.407, 39.898]], device=device) # bbox: [lon_min, lat_max, lon_max, lat_min] |
| |
| with torch.inference_mode(): |
| # Extract global embeddings |
| rs_embedding = model.encode_rs(rs, normalize=True) |
| sv_embedding = model.encode_sv(sv, normalize=True) |
| loc_embedding = model.encode_location(coords, normalize=True) |
| |
| # Query localized RS embeddings using the NILI module |
| rs_global, localized_rs_embedding = model.query_localized_rs( |
| rs, |
| coords, |
| bbox, |
| mode="nili", |
| normalize=True, |
| ) |
| ``` |
|
|
| ### Input Conventions |
| - **RS crop bbox order**: `[lon_min, lat_max, lon_max, lat_min]` |
| - **Coordinate order**: `[lon, lat]` |
| - **RS inputs**: Should be 10-band Sentinel-2 image (B2, B3, B4, B5, B6, B7, B8, B8A, B11, B12). |
|
|
| ## Citation |
|
|
| The citation entry will be added after the paper is officially published. For now, please refer to: |
|
|
| ```text |
| https://huggingface.co/papers/2503.16683 |
| ``` |
