| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: feature-extraction |
| tags: |
| - lunar |
| - remote-sensing |
| - multimodal |
| - hivit |
| - masked-autoencoder |
| - knowledge-distillation |
| --- |
| |
| # LANYUE-HiViT-Base |
|
|
| Six-modality LANYUE-HiViT-Base pretrained weights for lunar representation learning. |
| The model extends |
| the five-modality spherical-geolocation model with the LROC NAC branch through |
| three epochs of distillation from the epoch-12 teacher. |
|
|
| [Code](https://github.com/LLuKaoo/LANYUE) · |
| [Benchmark](https://huggingface.co/datasets/LLukaoo/LANYUE-bench) |
|
|
| ## Released checkpoint |
|
|
| | File | Description | |
| |---|---| |
| | [lanyue-hivit-base.pth](lanyue-hivit-base.pth) | Model tensors with embedded architecture configuration and preprocessing statistics. | |
| | [model_config.json](model_config.json) | Matching six-modality model configuration. | |
| | [preprocessing.json](preprocessing.json) | Saved pretraining normalization and signal-scaling rules. | |
| | [SHA256SUMS](SHA256SUMS) | Download integrity checksums. | |
|
|
| The PyTorch file supports `torch.load(..., weights_only=True)`. It contains |
| 678 model tensors, including the shared encoder and reconstruction |
| decoders. Optimizer/scaler state, machine-specific paths and unused legacy |
| modules are excluded. Retained tensors keep their original dtype and values. |
|
|
| ## Architecture and inputs |
|
|
| Use **`lanyue_nac`** with the model configuration stored in the checkpoint. |
| |
| | Model input key | Observation | Patch size | |
| |---|---|---:| |
| | `ce2_dom` | CE-2 DOM | 16 | |
| | `ce2_dem` | CE-2 DEM | 16 | |
| | `nasa_dom` | LROC WAC | 8 | |
| | `nasa_dem` | NASA lunar elevation | 8 | |
| | `minirf_sar` | Mini-RF SAR | 16 | |
| | `nac` | LROC NAC | 16 | |
| |
| The shared encoder has embedding dimension 512, stage depths 2/2/20 and |
| eight attention heads. Each reconstruction decoder has four blocks. |
| Dense feature extraction accepts any available subset, including NAC alone. |
| Arrays must be `[B,1,H,W]` with dimensions divisible by their patch size. |
| Physical scales are recorded in `model_config.json`; adjust |
| `model.modality_physical_scales` when the task's pixel scale differs. |
| |
| ## Download and load |
| |
| **Weight access requires manual approval.** Sign in to Hugging Face and |
| [request access](https://huggingface.co/LLukaoo/LANYUE) with the account |
| you will use for downloading. After approval, run `hf auth login` with |
| a read token from that account before downloading the weights. |
| |
| Clone the code and install its model dependencies: |
| |
| ```bash |
| git clone https://github.com/LLuKaoo/LANYUE.git |
| cd LANYUE/LANYUE |
| python -m pip install -r requirements.txt |
| python -m pip install huggingface_hub |
| hf auth login # After your access request is approved |
| ``` |
| |
| Run the following from that model directory: |
| |
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from lanyue.models import lanyue_nac |
| from lanyue.features import encode_features |
| |
| path = hf_hub_download("LLukaoo/LANYUE", "lanyue-hivit-base.pth") |
| checkpoint = torch.load(path, map_location="cpu", weights_only=True) |
| model = lanyue_nac(**checkpoint["config"]["model"]) |
| model.load_state_dict(checkpoint["model"], strict=True) |
| model.eval() |
|
|
| # Illustrative normalized inputs; apply the appropriate image preprocessing. |
| inputs = { |
| "nasa_dom": torch.randn(1, 1, 48, 88), |
| "nasa_dem": torch.randn(1, 1, 80, 144), |
| } |
| with torch.no_grad(): |
| tokens = encode_features(model, inputs) |
| descriptor = torch.cat([x.mean(1) for x in tokens.values()], dim=-1) |
| print(descriptor.shape) # [1, 1024] |
| ``` |
| |
| Coordinates can be supplied as `inputs["center_lonlat"]`, in longitude/latitude |
| order and degrees. Missing coordinates use the learned missing-location vector. |
| The feature API does not automatically normalize or resize observations. |
| Use the saved pretraining statistics for matching source imagery, and the |
| task configuration when reproducing downstream experiments. Task prediction |
| heads are trained separately and are not bundled in this pretrained checkpoint. |
|
|
| ## License and citation |
|
|
| [MIT](LICENSE). Source-code attribution is retained in [NOTICE](NOTICE). |
| Citation details will be added with the paper release. |
|
|
