EarthVL-pretrained
Pretrained weights of the two VQA branches of EarthVLNet, the method of the paper EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework (IEEE TPAMI, to be published).
- Code: https://github.com/Junjue-Wang/EarthVL
- Paper: https://arxiv.org/pdf/2601.02783
- Dataset: Kingdrone-Junjue/EarthVLSet
Stage-1 checkpoint required. Both weight files were trained on the features of the authors' stage-1 Semantic-FPN (ConvNeXt-L) checkpoint, which is not available yet; with the features of a re-trained stage 1, these weights are not usable (see Stage-1 checkpoint). Without the authors' features, the weights have not been evaluated with the GitHub code. The LLM-branch weights also come from an August-2024 run, later than the model behind the LLM-branch results of the paper, so their results can differ from the tables of the paper.
Files
| File | Size | SHA-256 | Content |
|---|---|---|---|
earthvlnet_llm_lora_40000.safetensors |
368,097,616 bytes | d7376bd526533f815192126984ffe034d4ec1de887ceaf3f2ae0372ae0d82b78 |
LLM branch (configs/EarthVLNet.py), step 40,000. Trainable parameters only: 462 bf16 tensors, 184.0M parameters. LoRA (r=64, alpha=16) on every linear layer of the language model except lm_head (448 tensors), object-guided attention oga (2), multi-modal projector mm_projector (4) and mask embedding mask_emb (8). The frozen language model is not included. |
earthvl_count_40000.pth |
87,039,843 bytes | 03ee128153a02abed4f3a0e93c0aebcc5b26eaa8558b529837edd6a1e2a88b8f |
Counting branch (configs/EarthVLCount.py), step 40,000. Plain PyTorch state dict: 53 float32 tensors, 21.8M parameters, no optimizer state. |
SHA256SUMS |
Checksums of the two files (sha256sum -c SHA256SUMS). |
The tensor names are those of EarthVLNet and EarthVLCount in the GitHub code. Both files load with
torch.load(weights_only=True) or safetensors; no full unpickling is needed.
How to use
With the GitHub code (installation, data and stage-1 features are described in its README):
# weights
huggingface-cli download Kingdrone-Junjue/EarthVL-pretrained --local-dir ./pretrain/EarthVL-pretrained
# frozen language model of the LLM branch: the Vicuna-7B language model of LLaVA-1.5 and its processor
python tools/prepare_vicuna.py --src llava-hf/llava-1.5-7b-hf --dst ./pretrain/vicuna-7b
# predictions on the test set: ./log/EarthVLNet/prediction-40000.json and ./log/EarthVLCount/prediction-40000.json
bash scripts/eval_cls.sh ./pretrain/EarthVL-pretrained/earthvlnet_llm_lora_40000.safetensors
bash scripts/eval_count.sh ./pretrain/EarthVL-pretrained/earthvl_count_40000.pth
# merge the two branches (counting answers from the counting branch) and score
python evaluate.py
eval_cls.sh loads the base weights from ./pretrain/vicuna-7b and the LoRA, OGA, MMP and mask-embedding weights
from the safetensors file. evaluate.py scores the test QAs that have an answer (EarthVL-GLOBAL Test). The EarthVQA
Test answers are withheld; predictions for them are scored by the
EarthVQA VQA challenge on Codabench.
Stage-1 checkpoint
Not available yet. Both branches read the stage-1 outputs of every image: the 1536-channel feature map of
the last ConvNeXt-L stage and the predicted land-cover mask (scripts/extract_features.sh). The released weights were
trained on the outputs of the authors' own Semantic-FPN (ConvNeXt-L) checkpoint, which is not available yet. A stage-1
network trained with the GitHub code (scripts/train_seg.sh) produces features in a different feature space; with
such features, train the two VQA branches yourself (scripts/train_count.sh, scripts/train_generation.sh) instead
of using these weights.
Training setup
Both runs used the Train splits of EarthVQA and EarthVL-GLOBAL without the Rotterdam area (images 9118-9485 of
EarthVL-GLOBAL/Train, which the paper assigns to Val), with random flips and 90-degree rotations of the features and
masks (direction answers rewritten accordingly) and random channel dropping. The GitHub configs use the settings of
both runs.
- LLM branch (August 2024): the Vicuna-7B language model of LLaVA-1.5
(llava-hf/llava-1.5-7b-hf) in bf16, frozen and without 4-bit
quantization, with LoRA (r=64, alpha=16, dropout 0.05). The stage-1 features and a 128-channel embedding of the
predicted mask give 1024 visual tokens through OGA and MMP. Trained on six question types (Basic Judging,
Reasoning-based Judging, Object Situation Analysis, Distribution Analysis, Direction Analysis, Comprehensive
Analysis) with channel drop rate 0.2. DeepSpeed 0.14.0 with ZeRO-2 and bf16 on 4 GPUs, micro batch 6 (batch 24);
Adam in AdamW mode (lr 2e-4, betas (0.8, 0.999), weight decay 3e-7) with
WarmupCosineLR; 40k steps. - Counting branch: a 3-layer Transformer (hidden size 384, 4 heads) on the same visual encoding and the question
(FLAN-T5 tokenizer,
google/flan-t5-xl), trained with the numerical difference loss on Basic Counting, Reasoning-based Counting and the counting questions of Comprehensive Analysis, with channel drop rate 0.1. AdamW (lr 1e-3, betas (0.9, 0.999), weight decay 0.2) with a cosine schedule and a 1,000-step warm-up; batch 32 on one GPU, fp32; 40k steps.
License
The weights are licensed under CC BY-NC-SA 4.0, like the code. They are used together with the Vicuna-7B language model of LLaVA-1.5, which is based on Llama 2: the Llama 2 Community License (and the terms of LLaVA and Vicuna) applies to that base model whenever these weights are used with it. The weights were trained on EarthVLSet, which is licensed under CC BY-NC-ND 4.0.
Citation
@article{wang2026earthvl,
title={EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework},
author={Wang, Junjue and Zhong, Yanfei and Chen, Zihang and Zheng, Zhuo and Ma, Ailong and Zhang, Liangpei},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2026},
note={to be published}
}
@article{wang2024earthvqa,
title={EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering},
author={Wang, Junjue and Zheng, Zhuo and Chen, Zihang and Ma, Ailong and Zhong, Yanfei},
journal={Proceedings of the AAAI Conference on Artificial Intelligence},
volume={38},
number={6},
pages={5481-5489},
year={2024},
DOI={10.1609/aaai.v38i6.28357}
}
Model tree for Kingdrone-Junjue/EarthVL-pretrained
Base model
llava-hf/llava-1.5-7b-hf