EO-Jev: Decision Model for Earth Observation

Paper (arXiv:2611.03371) · Project page · Code

EO-Jev answers questions about satellite and aerial images with a probability for every allowed answer, in a single forward pass. It writes no text, so answers come fast and with a confidence you can threshold: accept confident answers automatically and send uncertain ones to an analyst.

EO-Jev is d3-flash (8.4B parameters, built on Qwen3.5-9B) adapted to Earth observation. We trained LoRA adapters on public Earth-observation training splits: rank 32 on the text tower, 64 on the vision merger and 16 on the vision blocks. The adapters are merged into the weights, so EO-Jev has the same architecture, interface and inference cost as d3-flash.

Results

On 30 evaluation settings from 23 public Earth-observation benchmarks (details in the paper):


26 / 30	settings improved significantly over d3-flash: +23.8 points on average where the dataset's training split was used, +8.0 on 12 settings held out from adaptation
20 / 28	settings at or above the best open Earth-observation specialist model
21 / 28	settings with lower expected calibration error than the specialist
6.4×	median throughput of the best specialist's, each at its best batch size (3.2× with one question per call)
−1.8	points on general (non-EO) decisions, the cost of adaptation

Quickstart

pip install "transformers==5.17.0" torch torchvision pillow safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers
pip install causal-conv1d           # optional: fast kernel for the short convolutions in those layers

import json

from huggingface_hub import hf_hub_download
from transformers import AutoModel

model = AutoModel.from_pretrained("mx297/EO-Jev", trust_remote_code=True)  # on cuda:0 when a GPU is visible

image = hf_hub_download("mx297/EO-Jev", "assets/example_airport.png")  # or a local path, URL or PIL image
result = model.system_one(
    state="Aerial image, about 0.3 m per pixel.",
    images=[image],
    questions={
        "scene": {
            "type": "choice",
            "instructions": "What is the main land use in this image?",
            "criteria": {"airport": None, "harbor": None, "parking lot": None, "residential area": None, "farmland": None},
        },
        "airplanes": {
            "type": "choice",
            "instructions": "How many airplanes are in this image?",
            "criteria": {"0": None, "1": None, "2": None, "3": None, "4": None, "5": None, "more than 5": None},
        },
        "car": {
            "type": "noul",
            "instructions": "Is there a car in this image?",
        },
    },
)
print(json.dumps(result["answers"], indent=1))

Output (probabilities rounded to three decimals):

{
 "scene": {"type": "choice", "choice": "airport",
           "probabilities": {"airport": 0.966, "harbor": 0.000, "parking lot": 0.033, "residential area": 0.000, "farmland": 0.000},
           "confidence": 0.905},
 "airplanes": {"type": "choice", "choice": "5",
               "probabilities": {"0": 0.000, "1": 0.001, "2": 0.002, "3": 0.046, "4": 0.440, "5": 0.454, "more than 5": 0.057},
               "confidence": 0.463},
 "car": {"type": "noul", "noul": 0.978}
}

The scene and the car are answered right and with confidence. The image has four airplanes: EO-Jev's count is a near tie between 4 (0.440) and 5 (0.454), and its low confidence (0.46) marks the answer for review instead of hiding the mistake. Counting on unfamiliar imagery is one of the limitations below. For noul questions, the value is the probability of "yes".

All questions about the same image or images are answered in one call.

Asking questions

A request has a state (free text: what the image is, its sensor or resolution, any context), one or more images, and a dictionary of questions. Each question has a type:

Type	`criteria`	Answer
`choice`	a dict of options (up to 255); each value is an optional description (`None` is fine)	`choice` (the top option), `probabilities` per option, `confidence`
`noul`	none (a yes/no question)	`noul`: the probability of "yes"
`score`	an ordered list of 2–10 levels, e.g. `["no damage", "minor damage", "major damage", "destroyed"]`	`score` (the expected level index) and `probabilities` per level

Several images go into one request, for example a pre- and post-event pair or a time series. Say in the state which is which:

result = model.system_one(
    state="Image 1 is the earlier (pre-disaster) image and Image 2 the later (post-disaster) image of the same area.",
    images=["pre.png", "post.png"],
    questions={
        "damage": {
            "type": "score",
            "instructions": "How damaged is the building in the centre?",
            "criteria": ["no damage", "minor damage", "major damage", "destroyed"],
        }
    },
)

The model also works as a transformers pipeline: pipeline("decision", model="mx297/EO-Jev", trust_remote_code=True)(state=..., questions=..., images=...).

Tips

Keep each answer's wording short and distinct. The probabilities are over the options you give, so include an option such as "none of these" or "0" when it can be the answer.
The probabilities are calibrated at the checkpoint's temperature. A per-question threshold on the top probability decides which answers to accept automatically.
The code that runs the model (d3_runtime.py and the other d3_*.py files) is d3-flash's runtime, unchanged.

Training

Base model: d3-flash, with LoRA adapters on the text tower, the vision merger and the vision blocks (the vision blocks at 0.2× the learning rate), merged after training.
Data: decision questions built from public Earth-observation training splits: scene and land-use labels, object presence and counting, visual question answering, multi-temporal change, building damage and disaster type, geo-localisation. About 20% of every batch replays general decision data with d3-flash's own probabilities as targets, to keep its general skills. Every training image that duplicates or crops an evaluation image was removed.
Loss: KL divergence to the target distribution: one-hot, soft ordinal labels (SORD) for counts and damage levels, or d3-flash's own probabilities.
Compute: 20,000 steps of 16 questions on 4 NVIDIA A100 GPUs.

Training and evaluation code: github.com/mbzuai-oryx/EO-Jev.

Limitations

Counting on unfamiliar imagery, robustness to the order of answer options, geometric and cross-view reasoning, and calibration far from the training data remain the hardest cases.
EO-Jev was trained and evaluated on optical (RGB) imagery; SAR and multispectral inputs were not part of training.
It answers only with the options you provide. It cannot say that none applies unless such an option is offered.
Validate it on your own data before relying on it for decisions with consequences.

License

Apache-2.0 (LICENSE), as d3-flash, which is built on Qwen/Qwen3.5-9B (Apache-2.0); see NOTICE. The datasets used for training keep their own licences. The example image is from the UC Merced Land Use dataset (USGS National Map imagery, public domain).

Citation

@article{abouelhadid2026eojev,
  title   = {{EO-Jev}: Decision Model for {Earth} Observation},
  author  = {Abouelhadid, Mohamed Atef and Abuhani, Diaa Addeen and
             Munir, Muhammad Akhtar and Khan, Fahad Shahbaz and Khan, Salman},
  journal = {arXiv preprint arXiv:2611.03371},
  year    = {2026}
}

Please also cite d3-flash:

@misc{d3_flash_2026,
  title        = {{d3-flash}: A Multimodal Foundation Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-flash}}
}