abdevd/FOTBCD-CC-Qwen3-VL-8B-LoRA

FOTBCD-CC captioning adapter for Qwen3-VL-8B

This repository contains the LoRA adapter used to generate the FOTBCD-CC change captions. It adapts Qwen3-VL-8B-Instruct to compare two aligned aerial images of the same location, decide whether they show a lasting physical change, and return concise descriptions as structured JSON.

The first image must be the BEFORE (earlier) observation and the second image the AFTER (later) observation.

Output format

{
  "durable_change": true,
  "captions": [
    "A residential building with a dark roof now occupies the previously open field."
  ]
}

durable_change indicates whether the pair shows a persistent physical change. Changed pairs may contain several captions when several distinct events are visible. When it is false, the model returns one caption describing persistent features of the scene.

Usage

Install the required libraries:

pip install "transformers>=4.57" "peft>=0.19.1" accelerate torch pillow

Load the base model and adapter:

import torch
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

BASE_MODEL = "Qwen/Qwen3-VL-8B-Instruct"
ADAPTER = "abdevd/FOTBCD-CC-Qwen3-VL-8B-LoRA"

prompt = (
    "The first image is BEFORE (earlier) and the second image is AFTER (later). "
    "Compare the same location and return JSON only, with exactly a durable_change "
    "boolean and a captions array containing concise, scene-specific observations."
)

processor = AutoProcessor.from_pretrained(BASE_MODEL)
base_model = Qwen3VLForConditionalGeneration.from_pretrained(
    BASE_MODEL,
    dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER).eval()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "before.png"},
            {"type": "image", "image": "after.png"},
            {"type": "text", "text": prompt},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=64, do_sample=False)

generated = generated[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Training

Qwen3.8-Max annotated a selected set of 19,997 FOTBCD image pairs, producing 29,447 captions. These outputs provided supervised response-distillation targets for the Qwen3-VL-8B student.

The teacher set was divided deterministically with seed 42:

Split	Pairs
Training	15,997
Validation	2,000
Test	2,000

The adapter was trained for one epoch over the complete training split. A per-device batch size of 1 and 16-step gradient accumulation produced an effective batch size of 16 and 1,000 optimizer updates.

Setting	Value
Framework	ms-swift 4.5.2 / PEFT 0.19.1
Precision	bfloat16 training; float32 adapter weights
LoRA rank	8
LoRA alpha	32
LoRA dropout	0.05
Target modules	Language-model attention and MLP linear projections
Vision tower	Frozen
Visual aligner	Frozen
Learning rate	1e-4
Scheduler	Cosine with 5% warmup
Weight decay	0.1
Maximum sequence length	4,096

The released adapter contains 21,823,488 parameters. The base model weights are not included.

Evaluation

The adapter and unmodified base model were evaluated on the same held-out 2,000-pair test split. References were generated by Qwen3.8-Max.

Model	Valid JSON schema	Durable-change accuracy	Durable-change F1	Caption-count accuracy	Token F1	ROUGE-L
LoRA adapter	100.00%	89.60%	91.33%	79.20%	0.496	0.432
Base Qwen3-VL-8B	100.00%	62.35%	62.44%	11.95%	0.142	0.112

Caption scores use optimal order-independent matching between generated and reference captions. These results measure agreement with the teacher, not independently verified visual correctness.

Limitations

• The adapter was developed for aligned, earlier/later aerial-image pairs and has not been evaluated as a general-purpose vision-language adapter.
• Its definition of change emphasizes persistent physical changes. Seasonal appearance, cultivation, vehicles, shadows, and other temporary differences should not normally produce a positive decision.
• Teacher errors can be inherited by the student. Outputs should be visually reviewed before use as ground truth.
• Performance was measured on FOTBCD-derived imagery; behavior may differ across regions, sensors, resolutions, or acquisition conditions.

Related datasets

• FOTBCD: the underlying before/after aerial image dataset.
• FOTBCD-CC: 321,200 generated captions for 223,675 FOTBCD pairs.

The adapter is released under Apache 2.0, matching the base model. Dataset licenses are documented separately in their dataset cards.