| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-0.8B |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| language: |
| - en |
| tags: |
| - remote-sensing |
| - sentinel-2 |
| - multispectral |
| - earth-observation |
| - visual-question-answering |
| - calibration |
| - decision-model |
| - system-one |
| datasets: |
| - BIFOLD-BigEarthNetv2-0/BigEarthNet.txt |
| --- |
| |
| # EarthJev: typed answers about multispectral Sentinel-2 patches, without generating text |
|
|
| **Domenico Benfenati, Antonio Maria Rinaldi, Cristian Tommasino** — Department of Electrical Engineering and |
| Information Technology (DIETI), University of Naples Federico II. |
| Code: [https://github.com/CosmoIknosLab/EarthJev](https://github.com/CosmoIknosLab/EarthJev) · Paper: [arXiv:2611.02096](https://arxiv.org/abs/2611.02096). |
|
|
| <p align="center"><img src="assets/earthjev_framework.png" width="100%" alt="EarthJev framework"></p> |
|
|
| EarthJev answers typed questions about a Sentinel-2 L2A patch from **all 12 bands**. Each question declares its |
| options (yes/no, one of *K* options, an ordered level), and the model returns a **probability for every option**, |
| read from the logits of the option letters at a single position: no token is generated, so every answer is a valid |
| option, and questions about the same patch share one encoding of the image. |
|
|
| - **Backbone:** [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), fully fine-tuned with the open |
| [OneJev](https://github.com/OmniJev/OneJev) recipe (cross-entropy + Brier on the option letters, options permuted, |
| one epoch, learning rate 5e-6). |
| - **12-band input:** the input layer of the vision encoder is widened from 3 to 12 channels (R, G, B kernels copied |
| onto B04, B03, B02; the other bands start at zero) and is the only trained part of the encoder. |
| - **This checkpoint:** seed 0 of three, chosen on validation accuracy (85.63% on 36,698 validation |
| questions). Temperatures per question type and number of options, fitted on the validation questions |
| (`calibration.json`): choice:15 1.027, choice:2 0.991, choice:4 1.136, choice:5 1.025, noul:2 1.024. |
|
|
| ## Results |
|
|
| All models trained on the same 70,890 CORINE questions from 5% of BigEarthNet v2.0 (three seeds) and timed in one |
| job on one H100 PCIe GPU (batch 1, bfloat16; each timer runs from the decoded image in memory to the answer). |
| CORINE: 36,028 test questions (mean ± std over seeds), ECE = mean of the per-seed ECEs (15 bins). RSVQAxBEN: 29,501 |
| presence questions written by other authors, never used for training. *ms / image*: all the questions of an image |
| (6.1 on average) in one request. |
|
|
| | Model | Params | Input | CORINE acc. (%) | ECE | RSVQAxBEN acc. (%) | ms / question | Slower than EarthJev | ms / image | Slower than EarthJev | |
| |---|---|---|---|---|---|---|---|---|---| |
| | **EarthJev (this model)** | **0.8B** | **12 bands** | **86.0 ± 0.1** | **0.006** | **81.0** | **19.9** | — | **42.8** | — | |
| | Qwen3.5-0.8B, generative | 0.8B | 12 bands | 86.8 ± 0.2 | 0.005 | 80.5 | 134.7 | 6.8× | 191.9 | 4.5× | |
| | Qwen3-VL-2B, LoRA | 2.1B | RGB | 86.6 ± 0.1 | 0.005 | 80.6 | 111.6 | 5.6× | 176.2 | 4.1× | |
| | InternVL3-2B, LoRA | 2.1B | RGB | 87.1 ± 0.0 | 0.006 | 81.2 | 84.5 | 4.3× | 157.3 | 3.7× | |
| | EarthDial-4B, LoRA | 4.2B | RGB | 89.5 ± 0.1 | 0.006 | 82.4 | 171.8 | 8.7× | 431.6 | 10.1× | |
| | ResNet-50 VQA* | 0.14B | 12 bands | 87.5 ± 0.2 | 0.089 | 79.4 | 12.1 | 0.6× | 14.9 | 0.3× | |
| | Swin-T VQA* | 0.15B | 12 bands | 86.5 ± 0.1 | 0.047 | 79.6 | 16.2 | 0.8× | 17.9 | 0.4× | |
| | MambaVision-T VQA* | 0.15B | 12 bands | 85.9 ± 0.3 | 0.064 | 78.4 | 15.3 | 0.8× | 18.6 | 0.4× | |
|
|
| \* Fixed-vocabulary VQA models: a reference for the cost of a model that answers only the questions and options it |
| was trained on. Replacing the nine non-RGB bands with their training mean costs EarthJev 4.5 points on CORINE. |
|
|
| ## Usage |
|
|
| Requirements: `transformers` with Qwen3.5 support, `torch`, and OneJev's engine |
| (`pip install "qev @ git+https://github.com/OmniJev/OneJev"`). From a clone of this repository: |
|
|
| ```python |
| import numpy as np |
| from earthjev_infer import EarthJev |
| |
| model = EarthJev.load(".") # this folder (downloaded with huggingface_hub.snapshot_download) |
| s2 = np.load("examples/S2A_MSIL2A_20170701T093031_N9999_R136_T35VPK_37_55.npy") # [12, 120, 120] uint16 |
| answers = model.predict(s2, { |
| "dominant": {"type": "choice", "instructions": "Which land-cover class covers most of this patch?", |
| "criteria": {"artificial": "Artificial surfaces.", "agricultural": "Agricultural areas.", |
| "forest": "Forest and seminatural areas.", "wetlands": "Wetlands.", |
| "water": "Water bodies."}}, |
| "urban": {"type": "noul", "instructions": "Is there any urban fabric in this patch?"}, |
| }) |
| print(answers["dominant"]) # a probability for every option |
| ``` |
|
|
| The input is an array `[12, H, W]` with the Sentinel-2 L2A bands B01 B02 B03 B04 B05 B06 B07 B08 B8A B09 B11 B12 at |
| 10 m (20 m and 60 m bands resampled by pixel repetition), digital numbers as distributed (reflectance × 10,000), as |
| in BigEarthNet v2.0. `examples/` holds two test patches (7 questions, 7 questions) with |
| their CORINE test questions and an RGB preview. |
|
|
| ## Intended use and limitations |
|
|
| - Research on question answering and decision models for remote sensing. Not validated for operational use. |
| - Trained on 11,893 patches of 120 × 120 pixels from ten European countries (2017–2018) and on questions generated |
| from CORINE Land Cover (minimum mapping unit 25 ha); accuracy outside this distribution, on other sizes, products |
| (e.g. L1C) or question types is not established. On BigEarthNet.txt questions, whose types differ from the training |
| questions, accuracy is about 60% on binary and 40% on four-option questions. |
| - Only questions with declared options: open requests ("describe this image") are outside the model's interface. |
|
|
| ## Training data and licenses |
|
|
| Weights: Apache-2.0 (base model Qwen3.5-0.8B, Apache-2.0). Training and test data: BigEarthNet v2.0 (reBEN) and |
| BigEarthNet.txt (CDLA-Permissive-1.0; labels derived from CORINE Land Cover, © European Union, Copernicus Land |
| Monitoring Service); RSVQAxBEN (CC-BY-4.0) for testing. The example patches come from the reBEN test split. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{benfenati2026earthjev, |
| title = {{EarthJev}: Calibrated, Non-Generative Answers to Natural-Language Questions about {Sentinel-2} Land Cover}, |
| author = {Benfenati, Domenico and Rinaldi, Antonio Maria and Tommasino, Cristian}, |
| journal = {arXiv preprint arXiv:2611.02096}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2611.02096} |
| } |
| ``` |
|
|
| This study was partially supported by the PNRR MUR project PE0000013-FAIR. |
|
|
