EarthVL-pretrained

Pretrained weights of the two VQA branches of EarthVLNet, the method of the paper EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework (IEEE TPAMI, to be published).

Stage-1 checkpoint required. Both weight files were trained on the features of the authors' stage-1 Semantic-FPN (ConvNeXt-L) checkpoint, which is not available yet; with the features of a re-trained stage 1, these weights are not usable (see Stage-1 checkpoint). Without the authors' features, the weights have not been evaluated with the GitHub code. The LLM-branch weights also come from an August-2024 run, later than the model behind the LLM-branch results of the paper, so their results can differ from the tables of the paper.

Files

File Size SHA-256 Content
earthvlnet_llm_lora_40000.safetensors 368,097,616 bytes d7376bd526533f815192126984ffe034d4ec1de887ceaf3f2ae0372ae0d82b78 LLM branch (configs/EarthVLNet.py), step 40,000. Trainable parameters only: 462 bf16 tensors, 184.0M parameters. LoRA (r=64, alpha=16) on every linear layer of the language model except lm_head (448 tensors), object-guided attention oga (2), multi-modal projector mm_projector (4) and mask embedding mask_emb (8). The frozen language model is not included.
earthvl_count_40000.pth 87,039,843 bytes 03ee128153a02abed4f3a0e93c0aebcc5b26eaa8558b529837edd6a1e2a88b8f Counting branch (configs/EarthVLCount.py), step 40,000. Plain PyTorch state dict: 53 float32 tensors, 21.8M parameters, no optimizer state.
SHA256SUMS Checksums of the two files (sha256sum -c SHA256SUMS).

The tensor names are those of EarthVLNet and EarthVLCount in the GitHub code. Both files load with torch.load(weights_only=True) or safetensors; no full unpickling is needed.

How to use

With the GitHub code (installation, data and stage-1 features are described in its README):

# weights
huggingface-cli download Kingdrone-Junjue/EarthVL-pretrained --local-dir ./pretrain/EarthVL-pretrained
# frozen language model of the LLM branch: the Vicuna-7B language model of LLaVA-1.5 and its processor
python tools/prepare_vicuna.py --src llava-hf/llava-1.5-7b-hf --dst ./pretrain/vicuna-7b

# predictions on the test set: ./log/EarthVLNet/prediction-40000.json and ./log/EarthVLCount/prediction-40000.json
bash scripts/eval_cls.sh ./pretrain/EarthVL-pretrained/earthvlnet_llm_lora_40000.safetensors
bash scripts/eval_count.sh ./pretrain/EarthVL-pretrained/earthvl_count_40000.pth
# merge the two branches (counting answers from the counting branch) and score
python evaluate.py

eval_cls.sh loads the base weights from ./pretrain/vicuna-7b and the LoRA, OGA, MMP and mask-embedding weights from the safetensors file. evaluate.py scores the test QAs that have an answer (EarthVL-GLOBAL Test). The EarthVQA Test answers are withheld; predictions for them are scored by the EarthVQA VQA challenge on Codabench.

Stage-1 checkpoint

Not available yet. Both branches read the stage-1 outputs of every image: the 1536-channel feature map of the last ConvNeXt-L stage and the predicted land-cover mask (scripts/extract_features.sh). The released weights were trained on the outputs of the authors' own Semantic-FPN (ConvNeXt-L) checkpoint, which is not available yet. A stage-1 network trained with the GitHub code (scripts/train_seg.sh) produces features in a different feature space; with such features, train the two VQA branches yourself (scripts/train_count.sh, scripts/train_generation.sh) instead of using these weights.

Training setup

Both runs used the Train splits of EarthVQA and EarthVL-GLOBAL without the Rotterdam area (images 9118-9485 of EarthVL-GLOBAL/Train, which the paper assigns to Val), with random flips and 90-degree rotations of the features and masks (direction answers rewritten accordingly) and random channel dropping. The GitHub configs use the settings of both runs.

  • LLM branch (August 2024): the Vicuna-7B language model of LLaVA-1.5 (llava-hf/llava-1.5-7b-hf) in bf16, frozen and without 4-bit quantization, with LoRA (r=64, alpha=16, dropout 0.05). The stage-1 features and a 128-channel embedding of the predicted mask give 1024 visual tokens through OGA and MMP. Trained on six question types (Basic Judging, Reasoning-based Judging, Object Situation Analysis, Distribution Analysis, Direction Analysis, Comprehensive Analysis) with channel drop rate 0.2. DeepSpeed 0.14.0 with ZeRO-2 and bf16 on 4 GPUs, micro batch 6 (batch 24); Adam in AdamW mode (lr 2e-4, betas (0.8, 0.999), weight decay 3e-7) with WarmupCosineLR; 40k steps.
  • Counting branch: a 3-layer Transformer (hidden size 384, 4 heads) on the same visual encoding and the question (FLAN-T5 tokenizer, google/flan-t5-xl), trained with the numerical difference loss on Basic Counting, Reasoning-based Counting and the counting questions of Comprehensive Analysis, with channel drop rate 0.1. AdamW (lr 1e-3, betas (0.9, 0.999), weight decay 0.2) with a cosine schedule and a 1,000-step warm-up; batch 32 on one GPU, fp32; 40k steps.

License

The weights are licensed under CC BY-NC-SA 4.0, like the code. They are used together with the Vicuna-7B language model of LLaVA-1.5, which is based on Llama 2: the Llama 2 Community License (and the terms of LLaVA and Vicuna) applies to that base model whenever these weights are used with it. The weights were trained on EarthVLSet, which is licensed under CC BY-NC-ND 4.0.

Citation

@article{wang2026earthvl,
  title={EarthVL: A Progressive Earth Vision-Language Understanding and Generation Framework},
  author={Wang, Junjue and Zhong, Yanfei and Chen, Zihang and Zheng, Zhuo and Ma, Ailong and Zhang, Liangpei},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2026},
  note={to be published}
}
@article{wang2024earthvqa,
  title={EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering},
  author={Wang, Junjue and Zheng, Zhuo and Chen, Zihang and Ma, Ailong and Zhong, Yanfei},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={6},
  pages={5481-5489},
  year={2024},
  DOI={10.1609/aaai.v38i6.28357}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kingdrone-Junjue/EarthVL-pretrained

Adapter
(196)
this model

Dataset used to train Kingdrone-Junjue/EarthVL-pretrained

Paper for Kingdrone-Junjue/EarthVL-pretrained