Blame for README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
library_name: transformers
pipeline_tag: image-text-to-text
language:
- en
tags:
- remote-sensing
- sentinel-2
- multispectral
- earth-observation
- visual-question-answering
- calibration
- decision-model
- system-one
datasets:
- BIFOLD-BigEarthNetv2-0/BigEarthNet.txt
---

# EarthJev: typed answers about multispectral Sentinel-2 patches, without generating text

**Domenico Benfenati, Antonio Maria Rinaldi, Cristian Tommasino** — Department of Electrical Engineering and
Information Technology (DIETI), University of Naples Federico II.
Code: [https://github.com/CosmoIknosLab/EarthJev](https://github.com/CosmoIknosLab/EarthJev) · Paper: [arXiv:2611.02096](https://arxiv.org/abs/2611.02096).

<p align="center"><img src="assets/earthjev_framework.png" width="100%" alt="EarthJev framework"></p>

EarthJev answers typed questions about a Sentinel-2 L2A patch from **all 12 bands**. Each question declares its
options (yes/no, one of *K* options, an ordered level), and the model returns a **probability for every option**,
read from the logits of the option letters at a single position: no token is generated, so every answer is a valid
option, and questions about the same patch share one encoding of the image.

- **Backbone:** [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), fully fine-tuned with the open
  [OneJev](https://github.com/OmniJev/OneJev) recipe (cross-entropy + Brier on the option letters, options permuted,
  one epoch, learning rate 5e-6).
- **12-band input:** the input layer of the vision encoder is widened from 3 to 12 channels (R, G, B kernels copied
  onto B04, B03, B02; the other bands start at zero) and is the only trained part of the encoder.
- **This checkpoint:** seed 0 of three, chosen on validation accuracy (85.63% on 36,698 validation
  questions). Temperatures per question type and number of options, fitted on the validation questions
  (`calibration.json`): choice:15 1.027, choice:2 0.991, choice:4 1.136, choice:5 1.025, noul:2 1.024.

## Results

All models trained on the same 70,890 CORINE questions from 5% of BigEarthNet v2.0 (three seeds) and timed in one
job on one H100 PCIe GPU (batch 1, bfloat16; each timer runs from the decoded image in memory to the answer).
CORINE: 36,028 test questions (mean ± std over seeds), ECE = mean of the per-seed ECEs (15 bins). RSVQAxBEN: 29,501
presence questions written by other authors, never used for training. *ms / image*: all the questions of an image
(6.1 on average) in one request.

| Model | Params | Input | CORINE acc. (%) | ECE | RSVQAxBEN acc. (%) | ms / question | Slower than EarthJev | ms / image | Slower than EarthJev |
|---|---|---|---|---|---|---|---|---|---|
| **EarthJev (this model)** | **0.8B** | **12 bands** | **86.0 ± 0.1** | **0.006** | **81.0** | **19.9** | — | **42.8** | — |
| Qwen3.5-0.8B, generative | 0.8B | 12 bands | 86.8 ± 0.2 | 0.005 | 80.5 | 134.7 | 6.8× | 191.9 | 4.5× |
| Qwen3-VL-2B, LoRA | 2.1B | RGB | 86.6 ± 0.1 | 0.005 | 80.6 | 111.6 | 5.6× | 176.2 | 4.1× |
| InternVL3-2B, LoRA | 2.1B | RGB | 87.1 ± 0.0 | 0.006 | 81.2 | 84.5 | 4.3× | 157.3 | 3.7× |
| EarthDial-4B, LoRA | 4.2B | RGB | 89.5 ± 0.1 | 0.006 | 82.4 | 171.8 | 8.7× | 431.6 | 10.1× |
| ResNet-50 VQA* | 0.14B | 12 bands | 87.5 ± 0.2 | 0.089 | 79.4 | 12.1 | 0.6× | 14.9 | 0.3× |
| Swin-T VQA* | 0.15B | 12 bands | 86.5 ± 0.1 | 0.047 | 79.6 | 16.2 | 0.8× | 17.9 | 0.4× |
| MambaVision-T VQA* | 0.15B | 12 bands | 85.9 ± 0.3 | 0.064 | 78.4 | 15.3 | 0.8× | 18.6 | 0.4× |

\* Fixed-vocabulary VQA models: a reference for the cost of a model that answers only the questions and options it
was trained on. Replacing the nine non-RGB bands with their training mean costs EarthJev 4.5 points on CORINE.

## Usage

Requirements: `transformers` with Qwen3.5 support, `torch`, and OneJev's engine
(`pip install "qev @ git+https://github.com/OmniJev/OneJev"`). From a clone of this repository:

```python
import numpy as np
from earthjev_infer import EarthJev

model = EarthJev.load(".")                       # this folder (downloaded with huggingface_hub.snapshot_download)
s2 = np.load("examples/S2A_MSIL2A_20170701T093031_N9999_R136_T35VPK_37_55.npy")   # [12, 120, 120] uint16
answers = model.predict(s2, {
    "dominant": {"type": "choice", "instructions": "Which land-cover class covers most of this patch?",
                 "criteria": {"artificial": "Artificial surfaces.", "agricultural": "Agricultural areas.",
                              "forest": "Forest and seminatural areas.", "wetlands": "Wetlands.",
                              "water": "Water bodies."}},
    "urban": {"type": "noul", "instructions": "Is there any urban fabric in this patch?"},
})
print(answers["dominant"])   # a probability for every option
```

The input is an array `[12, H, W]` with the Sentinel-2 L2A bands B01 B02 B03 B04 B05 B06 B07 B08 B8A B09 B11 B12 at
10 m (20 m and 60 m bands resampled by pixel repetition), digital numbers as distributed (reflectance × 10,000), as
in BigEarthNet v2.0. `examples/` holds two test patches (7 questions, 7 questions) with
their CORINE test questions and an RGB preview.

## Intended use and limitations

- Research on question answering and decision models for remote sensing. Not validated for operational use.
- Trained on 11,893 patches of 120 × 120 pixels from ten European countries (2017–2018) and on questions generated
  from CORINE Land Cover (minimum mapping unit 25 ha); accuracy outside this distribution, on other sizes, products
  (e.g. L1C) or question types is not established. On BigEarthNet.txt questions, whose types differ from the training
  questions, accuracy is about 60% on binary and 40% on four-option questions.
- Only questions with declared options: open requests ("describe this image") are outside the model's interface.

## Training data and licenses

Weights: Apache-2.0 (base model Qwen3.5-0.8B, Apache-2.0). Training and test data: BigEarthNet v2.0 (reBEN) and
BigEarthNet.txt (CDLA-Permissive-1.0; labels derived from CORINE Land Cover, © European Union, Copernicus Land
Monitoring Service); RSVQAxBEN (CC-BY-4.0) for testing. The example patches come from the reBEN test split.

## Citation

```bibtex
@article{benfenati2026earthjev,
  title   = {{EarthJev}: Calibrated, Non-Generative Answers to Natural-Language Questions about {Sentinel-2} Land Cover},
  author  = {Benfenati, Domenico and Rinaldi, Antonio Maria and Tommasino, Cristian},
  journal = {arXiv preprint arXiv:2611.02096},
  year    = {2026},
  url     = {https://arxiv.org/abs/2611.02096}
}
```

This study was partially supported by the PNRR MUR project PE0000013-FAIR.