decider-12b-vision / README.md
Mapika's picture
decider-12b-vision v1: Gemma-4-12B-it + merged LoRA, text and image decisions
f695b8d verified
|
Raw History Blame Contribute Delete
8.39 kB
---
license: apache-2.0
base_model: google/gemma-4-12B-it
language: [en]
pipeline_tag: image-text-to-text
tags: [decision-model, calibrated, structured-output, system-one, one-pass, vision]
---
# decider-12b-vision: typed decisions with calibrated probabilities on Gemma-4-12B, from text and images, one pass
decider-12b-vision answers typed decisions with a probability for every option. It reads each question in one forward pass and
generates no text. The state may include images.
- **Input:** a state and one or more typed questions (Choice, Noul yes/no, Score), each with an explicit option list, in the
System One wire format (`POST /v1/systemone`). Images are sent as Jev-style `assets` and referenced from the state.
- **Output:** a probability distribution over the options of every question.
- **Weights:** [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) with a LoRA fine-tune (rank 32, attention
and MLP) merged into the weights, bf16, about 23 GB. The vision tower is the base model's, unchanged.
- **Code:** `decider.serve_vllm` in [decider-ai](https://github.com/Mapika/decider) 1.10.0 for text and images;
`decider.serve` and `decider.infer.Decider` for text.
## Which 12B to use
| | [decider-12b](https://huggingface.co/Mapika/decider-12b) v2 | decider-12b-vision |
|---|---|---|
| Decision Index 0.3 sample (4,500 requests) | 51.74 | **56.74** |
| Image questions, mean of 11 sets | 0.796 | **0.870** |
| JevBench public items, hard tier | 79 / 111 | **82 / 111** |
| Fresh held-out yes/no set, scored the JevBench way | **68.1** | 65.0 |
| Yes/no temperature in the config | 0.05 (JevBench's 0.2-0.8 rule) | fitted, calibrated |
decider-12b v2 stays the model we submit to JevBench: its yes/no answers score higher under JevBench's rule, and its config is
set for that board. decider-12b-vision is better on the Decision Index task mix, reads images, and gives calibrated yes/no
probabilities.
## Results
All numbers measured by us; none were verified by the benchmark maintainers.
| | stock Gemma-4-12B-it, our readout | decider-12b v2 | decider-12b-vision |
|---|---|---|---|
| Decision Index 0.3, 4,500-request sample | | 51.74 | **56.74** |
| Image questions, mean of 11 sets | 0.781 | 0.796 | **0.870** |
| Visual7W (not trained) | 0.880 | 0.880 | 0.883 |
| VSR (not trained) | 0.750 | 0.787 | 0.803 |
- The index sample is the 4,500-request sample of the 0.3 public suite; the full public suite was not run on this model. Through
the released server the sample gives 56.63.
- Image questions: the eval splits of 11 sets from [the_cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron),
300 items each. Visual7W and VSR were never trained on. Through the released server with the image sent as an asset:
Visual7W 0.880, VSR 0.800.
- Our own held-out sets against decider-12b v2: CLINC150 0.967 (0.903), iSarcasmEval 0.813 (0.707), VAST 0.793 (0.710), ACOS
0.987 / 0.853 (0.943 / 0.803), POP909-CL chords 0.690 (0.103), smart-home requests 0.998 (0.929).
- The temperatures were fitted on our own rows only; no Decision Index row, JevBench item or image evaluation item was used to
fit or choose them.
## Usage
```bash
python -m venv decider-vllm && . decider-vllm/bin/activate
pip install vllm==0.29.0 fastapi "uvicorn[standard]" jinja2 huggingface_hub pillow
pip install --no-deps "decider-ai>=1.10.0"
DECIDER_MODEL=Mapika/decider-12b-vision uvicorn decider.serve_vllm:app --host 127.0.0.1 --port 8000
```
A request with an image (the asset's key is the reference; the state names it):
```python
import base64, httpx
img = base64.b64encode(open("receipt.png", "rb").read()).decode()
r = httpx.post("http://127.0.0.1:8000/v1/systemone", timeout=60, json={
"state": {"receipt": "receipt-1", "policy": "Meals over 60 EUR need a manager's approval."},
"assets": {"receipt-1": {"type": "image", "format": "png", "data_base64": img}},
"questions": {"approval": {"type": "noul", "instructions": "Does the meal on `receipt` need approval?",
"criteria": {"true": "needs approval", "false": "no approval needed"}}}})
print(r.json()["answers"]["approval"])
```
Text-only use works the same without `assets`, and also through `decider.serve` or `Decider("Mapika/decider-12b-vision")`
(text only; those start without image input and say so).
## How a request is answered
1. **First reading.** Every question is rendered in the model's chat template with thinking off, and the log-probabilities of
the option letters are read at the answer slot. A softmax at T1(n) = 2.078 − 0.036 ln n (n = number of options) gives the
probabilities. Each image in the state is one placeholder, which vLLM expands to the image's soft tokens (at most 280).
2. **Second reading, only when unsure** (`decider.serve_vllm`): a question whose top probability is below 0.7 is read again with
its options reversed; the two log-softmaxes are averaged and a softmax at T2(n) = 2.025 − 0.042 ln n gives the final
probabilities.
## Training
Two LoRA stages (rank 32, alpha 64) on google/gemma-4-12B-it, then merged:
1. 955 steps at learning rate 5e-5 (28.0M tokens, 2.5 hours on one B300) on 18,000 text rows from the decider-31b training
mixture and 18,129 image rows.
2. 580 steps at 2e-5 (16.9M tokens) on the state-tracking mixture of decider-12b v2 (10,000 rows), plus 3,000 image rows and
4,000 text rows of stage 1 replayed.
- **Image rows:** train splits of seven the_cauldron subsets: AI2D, A-OKVQA, Hateful Memes, IconQA, RAVEN, ScienceQA, TQA.
Yes/no questions are trained as Noul questions, the others as Choice questions.
- **Text rows:** the decider-31b mixture (train splits of CLINC150, iSarcasmEval, VAST, ACOS, POP909-CL, GSM8K and NLI4CT,
decontaminated against every Decision Index 0.3 suite row; generated smart-home requests; generated state-tracking decisions
and decision families; document questions written by Qwen3.6-27B; labelled rows from public datasets).
- Not used: Decision Index test rows, JevBench or ImageJevBench items.
## Limitations
- **One pass only:** multi-step reasoning tasks are not solved by reading harder.
- **Images:** the gain is on the kinds of image questions trained; on new image question types it reads about as well as the
stock model. Box answers and image references in question text are not supported. Not measured on ImageJevBench.
- **Latency** was not measured for this model.
- **JevBench:** use decider-12b v2 for that board (see above).
## License
The weights are released under Apache 2.0, as the base model. Training sources and their terms:
- Image questions, from the train splits of [the_cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) subsets
(the_cauldron licenses only its own prompts, CC BY 4.0; each source keeps its own terms):
- AI2D (Kembhavi et al., 2016): CC BY-SA 4.0.
- A-OKVQA (Schwenk et al., 2022): Apache 2.0 (repository); images from COCO.
- IconQA (Lu et al., 2021): CC BY-NC-SA 4.0, non-commercial.
- ScienceQA (Lu et al., 2022): CC BY-NC-SA 4.0, non-commercial research use.
- TQA (Kembhavi et al., 2017): CC BY-NC 3.0, non-commercial.
- RAVEN (Zhang et al., 2019): no data licence stated (the code is GPL-3.0).
- Hateful Memes (Kiela et al., 2020): Facebook's Hateful Memes dataset licence. It permits use of the data to research and
develop models designed to detect manipulated media, and states that models derived from it may be used for academic and
commercial purposes. This is a research project; check the licence for your own use.
- CLINC150: CC BY 3.0 (Larson et al., 2019). Credit: "An Evaluation Dataset for Intent Classification and Out-of-Scope
Prediction", clinc/oos-eval.
- iSarcasmEval: MIT (Abu Farha et al., 2022). The texts are tweets.
- POP909-CL: MIT; built on POP909 (Wang et al., 2020), MIT.
- VAST (Allaway and McKeown, 2020) and ACOS (Cai et al., 2021): no licence stated by the authors.
- GSM8K (Cobbe et al., 2021): MIT.
- NLI4CT (SemEval-2024 Task 2, Jullien et al.): no licence file in the task repository; released for the shared task.
- RACE: non-commercial research use. It is a small part of the replayed data (250 rows).
- ARC, SNLI, LogiQA 2.0 (copy dependent): CC BY-SA 4.0; BoolQ: CC BY-SA 3.0; BANKING77, MedQA: CC BY 4.0; MMLU,
CommonsenseQA: MIT; MNLI: mixed per genre.