---
library_name: transformers
pipeline_tag: image-segmentation
base_model:
  - OpenGVLab/InternVL3_5-8B-HF
  - facebook/sam3
tags:
  - remote-sensing
  - reasoning-segmentation
  - earth-observation
  - custom_code
license: other  # TODO: SAM License on sam3_delta.safetensors, Apache-2.0 on the rest, some training datasets might be research use only
---

# MLRS

**More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe**

Accepted to **ACCV 2026**. 📄 Paper: https://arxiv.org/abs/2607.15942

MLRS is a remote-sensing vision-language model that answers questions and performs reasoning segmentation. 
MLRS supports wide range of inputs: ultra-high-res, SAR, thermal, false-color, multi-temporal, multi-view, visual prompting, multi-image. 
A LoRA-finetuned **InternVL3.5-8B** reasons about the image (or multiple images in rgb format) and either
answers in text or calls a segmentation tool:

```
<think>...</think><segmentation>{"noun phrase": "...", "objects": [{"bbox": [x1, y1, x2, y2]}]}</segmentation>
```

Each noun phrase, its boxes (on a 0–999 grid), and the last rgb image are passed to **SAM3**, whose mask decoder was finetuned
jointly via GRTO. SAM3 returns one mask per phrase, clipped to the predicted boxes. Text-only tasks
answer with `<answer>...</answer>` and return no masks.

This repository only holds the finetuned deltas (a LoRA adapter and SAM3 mask-decoder weights).
The base models are downloaded from the Hub at the revisions pinned in `config.json`.

> `facebook/sam3` is gated. Accept its license on the Hub and log in (`hf auth login` or set `HF_TOKEN` env.variable) before loading.

## Usage

```python
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

repo = "INSAIT-Institute/MLRS"
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)

image = Image.open("scene.png")
inputs = processor(images=image, text="Segment the buildings damaged by the flood.")
out = model.generate(**inputs, max_new_tokens=1024)

out.text[0]      # full generation, including <think> and the tool call / answer
out.phrases[0]   # noun phrases sent to SAM3
out.boxes[0]     # per phrase: pixel boxes [x1, y1, x2, y2]
masks = processor.post_process_masks(out, inputs)[0]  # bool [K, H, W], one mask per phrase
```

**Multiple images** (for example, pre/post-event pairs): `processor(images=[pre, post], text=...)`.
The segmentation tool runs on the last image; use `primary_image_index=` to pick a different one.

**Batching:** `processor(images=[[img_a], [img_b1, img_b2]], text=[q_a, q_b])`.

## Inputs and outputs

| Processor output | Description |
|---|---|
| `input_ids`, `attention_mask`, `pixel_values` | InternVL3.5 chat inputs. The instruction prompt is added automatically. |
| `sam_inputs` | SAM3 image inputs for the primary image |
| `sam_original_sizes` | `(H, W)` of the primary image after downsampling (longest side ≤ 4030, or ≤ 448 with 3 or more images) |
| `image_original_sizes` | `(H, W)` of the primary image as given |

`model.generate` returns `MLRSOutput(text, masks, phrases, boxes)`. `masks[i]` is a bool tensor
of shape `[K, H, W]` at `sam_original_sizes` resolution. `processor.post_process_masks` resizes the masks
back to the size of the input image.

## Notes

- The defaults reproduce the evaluation setup: greedy decoding, `max_new_tokens=1024`, and the
  LoRA adapter kept unmerged. `AutoModel.from_pretrained(..., merge_lora=True)` generates faster, with small numerical differences.
- Tested with `transformers==5.18.0`, `peft==0.18.0`, `torch==2.9.0`.

## Citation

```bibtex
@misc{ailuro2026mlrs,
      title={{More with Less}: a Large Scale Remote Sensing {VLM} with a Simple Recipe}, 
      author={Stefan Maria Ailuro and Mario Markov and Mohammad Mahdi and Luc Van Gool and Danda Pani Paudel},
      year={2026},
      eprint={2607.15942},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.15942}, 
}
```