Add V11 model card and updated Python image API examples

README.md

@@ -0,0 +1,228 @@
+---
+language:
+- en
+library_name: devision
+pipeline_tag: visual-question-answering
+base_model:
+- convaiinnovations/laya
+- google/siglip2-base-patch16-256
+tags:
+- devision
+- system-one
+- calibrated-decisions
+- rlcd
+- vision
+- jev
+---
+
+# deVision V11
+
+**Image + typed questions → structured answers and calibrated probabilities.** deVision is a non-autoregressive visual decision model built from Laya's ModernBERT-large decision model and a SigLIP2 vision encoder. It scores the supplied options without generating text. Multiple questions about one image share a single image encoding.
+
+This card describes **V11 (`v11-v9b-recovery`)**, selected by the project owner on October 6, 2026. It supports English, one image per request, yes/no decisions (`noul`) and multiple-choice decisions (`choice`).
+
+**Repository status, October 6, 2026:** this repository currently contains the model card; the V11 weights and configuration files have not yet been uploaded. Loading `lukbit/devision` will work after those files are added. With a local checkout and checkpoint, use `runs/v11-v9b-recovery/stage2` instead.
+
+## Install
+
+```bash
+pip install "devision @ git+https://github.com/byebyebruce/devision"
+```
+
+Python 3.11 or newer is required. For HTTP serving and the browser demo, install `devision[serve]` instead. If the model repository is private, authenticate with `hf auth login` using an account with access.
+
+The Python examples below require the new `image=` API (code commit `f34e81a` or later). As of October 6, the GitHub default branch has not yet received this change. Until it does, install from an updated local deVision checkout with `pip install .`; installing from GitHub currently provides the legacy `state` image-part API.
+
+## Python quickstart
+
+The following Hub example applies once the weights are uploaded:
+
+```python
+import devision
+
+model = devision.load("lukbit/devision", device="cpu")
+result = model.predict(
+    image="photo.jpg",  # replace with your image path or an HTTP(S) image URL
+    state="",           # required; empty when there is no additional context
+    questions={
+        "has_fork": {
+            "type": "noul",
+            "instructions": "Is there a fork in the image?",
+        },
+        "room": {
+            "type": "choice",
+            "instructions": "Which room is this?",
+            "criteria": {"kitchen": None, "bathroom": None, "bedroom": None},
+        },
+    },
+)
+print(result["answers"]["has_fork"]["noul"])       # probability of yes
+print(result["answers"]["room"]["choice"])         # selected option
+print(result["answers"]["room"]["probabilities"])  # probability for each option
+```
+
+`model.decide(...)` accepts the same arguments. The default device is `auto` (CUDA, then MPS, then CPU); pass `device="cpu"`, `"mps"` or `"cuda"` to choose explicitly.
+
+### Image input
+
+Use the separate Python `image=` argument. It automatically accepts:
+
+| Input | Example |
+|---|---|
+| HTTP(S) URL | `image="https://example.com/photo.jpg"` — replace with a real image URL |
+| Local path | `image="photo.jpg"` or `image=Path("photo.jpg")` with `from pathlib import Path` |
+| Base64 data URI | `image="data:image/png;base64,..."` — supply the complete encoding |
+| Plain Base64 | `image=base64_string` |
+| Encoded image bytes | `image=image_bytes` — PNG/JPEG file contents, for example |
+| PIL image | `image=pil_image` |
+
+### Context and notes
+
+`state` is required and accepts a string, object or array. For image-only questions, pass `state=""`; do not omit it or pass `None`. To supply additional context:
+
+```python
+result = model.decide(
+    image="photo.jpg",
+    state={"note": "The customer says this item arrived damaged."},
+    questions={
+        "damaged": {
+            "type": "noul",
+            "instructions": "Is the item visibly damaged?",
+        },
+    },
+)
+```
+
+`note` is an ordinary context field, not a special parameter. An `image` key inside an object state also stays text: `state={"image": "photo.jpg"}` does not load the file. The legacy `state=[{"type": "image", "url": "https://..."}]` form remains supported, but do not combine it with `image=`.
+
+### Output
+
+- `noul` is the probability that the statement holds, between 0 and 1.
+- `choice` is the highest-probability option; `probabilities` contains every option and sums to 1.
+- Choice `confidence` follows Jev: `(n * p_max - 1) / (n - 1)`.
+- Invalid request formats or image encodings raise `devision.InvalidRequest`. Local file reads can also raise `OSError`.
+
+## HTTP API and browser demo
+
+Once the weights are uploaded:
+
+```bash
+pip install "devision[serve] @ git+https://github.com/byebyebruce/devision"
+devision-serve --checkpoint lukbit/devision --device cpu --port 8000
+```
+
+Open `http://127.0.0.1:8000` for the demo. To use a local V11 checkpoint now, replace `lukbit/devision` with `runs/v11-v9b-recovery/stage2`.
+
+HTTP keeps the Jev-compatible `/v1/systemone` format. The following example uploads a local image as Base64:
+
+```python
+import base64
+import json
+from pathlib import Path
+from urllib.request import Request, urlopen
+
+body = {
+    "state": [{
+        "type": "image",
+        "base64": base64.b64encode(Path("photo.jpg").read_bytes()).decode("ascii"),
+    }],
+    "questions": {
+        "has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"},
+    },
+}
+request = Request(
+    "http://127.0.0.1:8000/v1/systemone",
+    data=json.dumps(body).encode("utf-8"),
+    headers={"Content-Type": "application/json"},
+)
+with urlopen(request, timeout=30) as response:
+    print(json.load(response))
+```
+
+For a remote image, use `{"type": "image", "url": "https://..."}` instead. HTTP does not load images from a top-level `image` field or from server-local paths. An image part's `url` accepts HTTP(S); its `base64` accepts plain Base64 without a Data URI prefix. Invalid requests and unsupported `score` questions return HTTP 422.
+
+## Architecture
+
+| Component | V11 |
+|---|---|
+| Vision encoder | Frozen SigLIP2-B/16, 256 × 256 input, about 93M parameters |
+| Image preprocessing | Aspect-preserving resize and letterbox padding |
+| Connector | 2 × 2 patch grouping by channel concatenation, then LayerNorm → Linear → GELU → Linear |
+| Visual sequence | 256 patch features become 64 visual tokens, inserted after `[CLS]` |
+| Text encoder | Laya-initialized ModernBERT-large, bidirectional, about 395M parameters |
+| Decision head | Laya's two Transformer layers and option scorer, about 27M parameters |
+| Checkpoint | About 518M parameters, FP32, about 2.07 GB of weights |
+
+Each option is scored at its `[MASK]` position. Calibrated softmax converts option scores into probabilities. The checkpoint contains merged LoRA weights; a separate adapter is not required for inference. Load it with `devision.load`, which understands the custom checkpoint layout.
+
+## Training and calibration
+
+V11 continues from V9B, following earlier image-caption alignment and RLCD decision training. The final round used **9,988 distinct questions for one epoch, 1,249 optimizer steps**, on Apple MPS in FP32:
+
+- 3,000 counting questions;
+- 4,500 replay questions, including ScienceQA, VSR and Visual7W;
+- 2,488 left/right questions retained from V9B.
+
+Learning rates were `5e-5` for the projector and decision head and `1e-4` for LoRA, with 125 warmup steps. See the [code and training log](https://github.com/byebyebruce/devision) for the full training lineage and configurations.
+
+Temperature scaling was fitted on the project's development/calibration mixture. Inference first uses the matching question-type/option-count bucket and otherwise falls back to the question-type temperature:
+
+| Temperature | Value |
+|---|---|
+| Choice, 2 options | 2.1514 |
+| Choice, 3–5 options | 1.9454 |
+| Choice fallback | 1.9879 |
+| Noul | 1.8161 |
+
+Calibration changes probabilities, not the model's underlying visual ability. Validate confidence thresholds on your own distribution.
+
+## Evaluation
+
+These are saved results from the V11 checkpoint, evaluated through `decide` on **MPS** with fitted temperatures. ECE uses 15 bins. The project test sets use held-out images; they are project-specific subsets or generated questions, not official leaderboard scores. Several benchmarks have also informed decisions across training rounds, so they should not be interpreted as untouched final tests throughout development.
+
+| Set | Questions | Accuracy | ECE |
+|---|---:|---:|---:|
+| COCO object presence | 1,120 | 0.955 | 0.022 |
+| VQAv2 multiple choice | 1,420 | 0.891 | 0.038 |
+| COCO size | 1,100 | 0.857 | 0.021 |
+| POPE, project filtered set | 8,676 | 0.844 | 0.072 |
+| GQA val subset | 992 | 0.745 | 0.085 |
+| COCO position | 1,274 | 0.850 | 0.013 |
+| VQAv2 yes/no subset | 1,000 | 0.716 | 0.043 |
+| COCO relative position | 1,950 | 0.651 | 0.033 |
+| VSR, project held-out split | 904 | 0.653 | 0.033 |
+| Visual7W, project held-out split | 1,000 | 0.712 | 0.040 |
+| Fresh counting test | 600 | 0.723 | 0.043 |
+
+### Comparison with Laya Vision 201M
+
+The following comparisons use Laya Vision's published per-question predictions on the same questions. Differences are percentage points, with 95% intervals from paired resampling by image. We did not rerun Laya Vision on our hardware.
+
+| Set | Questions | Laya Vision | deVision V11 | Difference, percentage points |
+|---|---:|---:|---:|---|
+| VQAv2 yes/no | 4,887 | 0.717 | 0.729 | +1.2 [−0.4, +2.7] |
+| A-OKVQA | 1,138 | 0.598 | 0.603 | +0.4 [−2.9, +3.7] |
+| ScienceQA with images | 2,097 | 0.824 | 0.747 | −7.7 [−9.8, −5.6] |
+
+The first two intervals include zero; these results do not establish superiority or equivalence. The full VQAv2 and A-OKVQA comparison sets include some images used in deVision training; the training log also reports subsets excluding those images. ScienceQA's compared images are unseen in training.
+
+On the full POPE random/popular/adversarial sets (3,000 questions each), V11 scores **0.863 / 0.849 / 0.813**, versus Laya Vision's published **0.836 / 0.819 / 0.777**. Only aggregate baseline scores are available for this comparison; it is not paired. These full sets differ from the 8,676-question filtered set above.
+
+## Limitations and model selection
+
+- **Counting recovery was not demonstrated.** On the fresh 600-question test, V11 differs from V9B by −0.3 percentage points [−2.7, +1.8], missing the predefined improvement target. The owner selected V11 as a tradeoff, particularly for improved left/right position consistency; this is not a claim that every acceptance target passed.
+- **Spatial reasoning is incomplete.** Correct answers on both an original image and its mirror occur for about 61% of left/right position pairs and 39% of relative-position choice pairs, but only 10% of relative-position yes/no pairs.
+- **Scientific diagrams remain difficult.** On 323 natural-science questions judged to require the image, V11 scores 0.467 versus Laya Vision's 0.700. On the full ScienceQA set, swapping in unrelated images still yields 0.665 accuracy, compared with 0.747 for the correct images; text and answer priors contribute substantially.
+- **Old-task protection is uncertain in two comparisons.** VSR and A-OKVQA did not establish the predefined lower bound of −2 percentage points relative to V9B. This is uncertainty about protection, not proof of a significant decline.
+- **Small text and fine detail are limited by 256 × 256 input.** The model is not a general-purpose OCR or text-generation system.
+- **English, single image, `noul` and `choice` only.** Other languages, multi-image input and `score` questions are not supported.
+- **Probabilities can be miscalibrated out of domain.** A low ECE on one benchmark does not guarantee reliable confidence elsewhere.
+
+## Links and intended use
+
+- [Code, training configurations and experiment records](https://github.com/byebyebruce/devision)
+- [Laya decision model](https://huggingface.co/convaiinnovations/laya)
+- [SigLIP2 vision encoder](https://huggingface.co/google/siglip2-base-patch16-256)
+- [Laya Vision reference implementation and published evaluations](https://github.com/r33drichards/laya-vision)
+
+The project is intended for non-commercial research. This card does not assign a new license to upstream models or datasets; their respective terms remain applicable.