Add model card

README.md

@@ -0,0 +1,285 @@
+---
+license: cc-by-nc-4.0
+base_model: Qwen/Qwen3-VL-2B-Instruct
+pipeline_tag: image-text-to-text
+library_name: transformers
+tags:
+  - vision-language
+  - image-classification
+  - calibrated-probabilities
+  - qwen3-vl
+---
+
+# Qevi-2B
+
+A full fine-tune of [Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) that
+answers **typed, closed questions about images** by reading the model's own logits instead of
+generating text.
+
+It is not a chat model. Ask it "is there a ladder in this image?" and it returns
+`P(Yes) = 0.97`, not a sentence. In exchange for giving up open-ended answers you get
+substantially better accuracy on closed questions, calibrated probabilities you can threshold on,
+and up to ~21x faster inference when asking many questions about one image.
+
+| | Base Qwen3-VL-2B | **Qevi-2B** |
+|---|---|---|
+| Accuracy, in-domain | 0.855 | **0.977** |
+| Accuracy, held-out domains | 0.745 | **0.889** |
+| Expected Calibration Error, held-out | 0.160 | **0.054** |
+
+---
+
+## Read this before you use it
+
+**This model must be used through a logit readout, not `.generate()`.** All the numbers above are
+measured by reading the LM-head logits at the position where the model would begin its reply,
+restricted to the allowed answer tokens. If you call `.generate()` and parse the text, you will not
+reproduce them, because that is not the path the model was fine-tuned on.
+
+The readout is ~30 lines of ordinary transformers code and needs no `trust_remote_code`. A
+self-contained version is below; the [`qevi`](#the-qevi-package) package adds question packing.
+
+---
+
+## Usage
+
+### Minimal, dependency-free (one question)
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
+
+REPO = "MeerDevelopment/Qevi-2B"
+model     = Qwen3VLForConditionalGeneration.from_pretrained(REPO, dtype=torch.bfloat16).cuda().eval()
+processor = AutoProcessor.from_pretrained(REPO)
+tok       = processor.tokenizer
+
+image     = Image.open("photo.jpg").convert("RGB")
+statement = "There is a ladder in this image."
+answers   = ["Yes", "No"]
+
+prompt = (
+    "<|im_start|>user\n<|vision_start|><|vision_end|>"
+    f"Statement: {statement}\n"
+    "Is this statement true of the image? Answer Yes or No."
+    "<|im_end|>\n<|im_start|>assistant\n"
+)
+
+inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
+with torch.no_grad():
+    logits = model(**inputs).logits[0, -1]          # scores at the sentinel position
+
+cand = [tok.encode(a, add_special_tokens=False)[0] for a in answers]
+probs = torch.softmax(logits[cand].float() / 2.45, dim=-1)   # 2.45 = calibration temperature
+print(dict(zip(answers, probs.tolist())))           # {'Yes': 0.97, 'No': 0.03}
+```
+
+### With the `qevi` package (many questions, one forward pass)
+
+```python
+from qevi import build_readouts, forward_logits, softmax, noul_question, choice_question_generic
+
+specs = [
+    (noul_question("There is a ladder in this image."),                    ["Yes", "No"]),
+    (noul_question("There is a bicycle in this image."),                   ["Yes", "No"]),
+    (choice_question_generic("Which is shown?", ["ladder", "bicycle",
+                                                 "car", "person"]),        ["A", "B", "C", "D"]),
+]
+readouts = build_readouts(tok, specs)
+logit_sets = forward_logits(model, processor, image, readouts)   # ONE forward pass
+
+for lg in logit_sets:
+    print([round(p, 4) for p in softmax(lg, T=2.45)])
+```
+
+All three questions share a single encoding of the image and are isolated from each other by a
+block-diagonal attention mask, so the answers are identical to asking them separately
+(verified bitwise, see [Order invariance](#order-invariance)).
+
+---
+
+## Question types
+
+| Type | Answer alphabet | Calibration temperature |
+|---|---|---|
+| `noul` | `["Yes", "No"]` | 2.45 |
+| `choice` | `["A", "B", ...]`, up to 26 options | 1.9 |
+| `score` | ordered level words, e.g. `["Poor","Fair","Good","Excellent"]` | 3.0 |
+
+Temperatures were fitted on held-out data; dividing logits by them before softmax is what makes
+the returned probabilities trustworthy (held-out ECE ~0.03-0.05 rather than ~0.16).
+
+---
+
+## Evaluation
+
+Scored against **base Qwen3-VL-2B-Instruct run through the identical readout path**, on the same
+images, in the same session. Both models therefore differ only in their weights, which is what
+isolates the effect of fine-tuning from the effect of the readout. Neither model saw any of these
+images during training.
+
+### In-domain (17,100 questions / 5,700 images, domains seen in training)
+
+| Metric | Base 2B | Qevi-2B | Δ |
+|---|---|---|---|
+| Accuracy | 0.855 | **0.977** | +0.122 |
+| Brier (lower better) | 0.231 | **0.039** | −0.192 |
+| ECE (lower better) | 0.088 | **0.025** | −0.063 |
+| Overconfidence gap | +0.088 | **−0.015** | −0.103 |
+
+### Held-out (10,800 questions / 3,600 images, 12 domains **never trained on**)
+
+| Metric | Base 2B | Qevi-2B | Δ |
+|---|---|---|---|
+| Accuracy | 0.745 | **0.889** | +0.144 |
+| Brier | 0.407 | **0.183** | −0.224 |
+| ECE | 0.160 | **0.054** | −0.106 |
+| Overconfidence gap | +0.160 | **+0.054** | −0.106 |
+
+The model improved **more** on unseen domains (+14.4) than on trained ones (+12.2), which is the
+main evidence that it learned a transferable skill rather than memorising the corpus.
+
+> **A note on the baseline.** The corpus stores `teacher_probs` recorded when it was built, and an
+> earlier draft of this card used those as the baseline. Three of the 31 groups (`food101`,
+> `flowers102`, and the content-safety set) had been labelled by the larger Qwen3-VL-**8B**, so
+> that baseline was a mixture of two models rather than the 2B this model was fine-tuned from.
+> The tables above instead come from re-running base 2B over both eval sets directly. The
+> difference is small (in-domain 0.855 vs the mixed 0.863; held-out 0.745 vs 0.741) and moves the
+> headline slightly in this model's favour, but the comparison is now exactly what it claims to be.
+
+### Generalisation to unseen corruptions
+
+Training included only `clean`, `contrast`, `gaussian_noise` and `impulse_noise`. The other four
+corruptions were held out entirely, and gained just as much:
+
+| Satellite imagery | Seen in training? | Base | Qevi-2B |
+|---|---|---|---|
+| clean / contrast / gaussian / impulse | yes | 0.54–0.72 | 0.96–0.98 |
+| jpeg / motion blur / pixelate / spatter | **no** | 0.45–0.61 | 0.89–0.97 |
+
+### Known regressions
+
+Two of 31 domains got **worse**. Stated plainly because they bound what this model is good for:
+
+- **German traffic signs (GTSRB): 0.834 vs 0.879 base.** The only true regression. No traffic-sign
+  data was in the training corpus, so nothing anchored this domain against weight drift. If signs
+  matter to you, use the base model or fine-tune further with sign data included.
+- **Pixelated car models: 0.474 vs 0.501 base.** Both models are near the floor. Distinguishing 196
+  car models after heavy pixelation is close to impossible; the information is not in the image.
+
+### Speed
+
+Packed inference vs. asking the same questions one at a time. Both paths read logits rather than
+generating, so this measures the benefit of packing alone and is not flattered by comparing
+against text generation. RTX 3090, bfloat16, CUDA-synchronised, median of 3 runs after warm-up:
+
+| Questions per image | 1 | 5 | 10 | 20 | 30 |
+|---|---|---|---|---|---|
+| Packed (ms) | 504 | 509 | 518 | 577 | 618 |
+| One at a time (ms) | 501 | 2,504 | 5,002 | 9,988 | 15,013 |
+| **Speedup** | 0.99x | **4.9x** | **9.7x** | **17.3x** | **24.3x** |
+
+Marginal cost of each additional question is ~5 ms on this hardware, against ~500 ms to encode the
+image; that ratio is the whole mechanism.
+
+**There is no benefit at N=1** (0.99x, i.e. parity), which is expected: packing one question is
+just asking one question. The gain comes entirely from amortising the image encode, so it grows
+with the number of questions you ask per image.
+
+Absolute latencies are image-dependent (a larger image means more vision tokens) and
+hardware-dependent; the speedup ratio is the portable number.
+
+### Order invariance
+
+Permuting question order yields bitwise-identical logits within float tolerance (atol 1e-5). On
+choice and score tasks, 90/90 images produced byte-for-byte identical predictions packed vs.
+unpacked. Packing does not silently change answers.
+
+---
+
+## Training
+
+| | |
+|---|---|
+| Base | Qwen3-VL-2B-Instruct (2,127,532,032 params, Apache 2.0) |
+| Method | Full fine-tune, all parameters trainable (no LoRA) |
+| Objective | Cross-entropy over candidate-answer logits only, label smoothing 0.05 |
+| Data | 85,500 questions over 28,500 images, 3 questions per image |
+| Epochs | 1 |
+| LR | 2e-5, OneCycle, 10% warm-up |
+| Batch | 8 images via gradient accumulation |
+| Optimiser | 8-bit AdamW (bitsandbytes), grad clip 1.0 |
+| Precision | bfloat16, gradient checkpointing enabled |
+| Hardware | 2x RTX 3090, ~2.5 hours |
+
+Training targeted **ground-truth labels** from the source datasets, not the base model's
+predictions, so the model learns to be correct rather than to imitate its teacher.
+
+### Training data
+
+Assembled from 31 publicly available image datasets spanning natural photos, sketches, paintings,
+clipart, satellite imagery, textures, fine-grained species and product categories, plus synthetic
+image corruptions. **The corpus itself is not redistributed with this model.**
+
+## Licensing
+
+**Weights: CC BY-NC 4.0 (non-commercial).** Not because the base model requires it, but because of
+training-data provenance. Of the twelve source dataset families, only EuroSAT and the
+content-safety set are permissively licensed (MIT); the other ten (CUB-200-2011, Stanford Cars,
+Food-101, SUN397, Oxford Flowers-102, Oxford-IIIT Pets, DTD, GTSRB, DomainNet, RESISC45) are
+distributed for academic or research use only. Whether a research-only dataset licence reaches the
+weights of a model trained on it is genuinely unsettled, so this release takes the cautious side
+rather than granting commercial rights it may not be in a position to grant.
+
+**Code: Apache 2.0.** The [`qevi`](#the-qevi-package) package carries no data-provenance
+encumbrance and is freely usable, including commercially.
+
+**Base model: Apache 2.0** ([Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct)),
+unaffected by the above.
+
+**If you need commercially-licensable weights:** the method, the code and the base model are all
+permissive. Retrain using the `qevi` package on data you own or that is licensed for commercial
+use, and the resulting weights carry no encumbrance from this corpus. The recipe is fully
+documented above (1 epoch, lr 2e-5, ~5 GPU-hours on two consumer cards).
+
+---
+
+## Limitations
+
+- **Closed answers only.** Every question needs a finite answer set declared up front. It cannot
+  caption, describe, or answer something you did not anticipate.
+- **Free-form generation survives, but answers get terser.** The fine-tune optimised only the
+  typed-readout objective, so we checked whether ordinary `.generate()` still works. It does:
+  across 5 held-out images x 3 open prompts, greedy decoding produced coherent, grammatical,
+  on-topic text with no repetition collapse (degeneracy 0.002 vs 0.001 for base) and no empty
+  outputs. The measurable change is **length: mean 28.3 generated tokens vs 50.7 for the base**,
+  a ~44% drop. Having been trained on one-word answers, the model answers open questions more
+  directly ("Spiders" where the base writes "The main object in this image is a spider.").
+  This is a behavioural shift, not a capability loss, but if you want verbose descriptions the
+  base model is the better choice.
+- **Answer tokens must be single-token-distinguishable.** Candidates are matched on their first
+  token, so options sharing a first token (e.g. "Poor" vs "Poorly") collide.
+- **English prompts only**, matching the prompt templates used in training.
+- **Not a safety classifier.** One training domain involved content-severity labels, but this model
+  has not been validated for moderation use and should not be deployed for it without its own
+  evaluation.
+
+## The `qevi` package
+
+Packing, prompt construction and the readout live in a small Apache-2.0 package
+(~350 lines, torch only):
+
+```bash
+pip install qevi
+```
+
+## Citation
+
+```bibtex
+@misc{qevi2b,
+  title  = {Qevi-2B: typed question answering over images without text generation},
+  year   = {2026},
+  note   = {Full fine-tune of Qwen3-VL-2B-Instruct with packed native-logit readout}
+}
+```