|
Download README.md from Mapika/decider-12b-vision: direct link, hf CLI and curl.
- Browser
- Download file 8.39 kB
-
https://huggingface.co/Mapika/decider-12b-vision/resolve/main/README.md
- Command line
-
hf download hf://Mapika/decider-12b-vision/README.md
-
curl -L -o README.md https://huggingface.co/Mapika/decider-12b-vision/resolve/main/README.md
8.39 kB
| license: apache-2.0 | |
| base_model: google/gemma-4-12B-it | |
| language: [en] | |
| pipeline_tag: image-text-to-text | |
| tags: [decision-model, calibrated, structured-output, system-one, one-pass, vision] | |
| # decider-12b-vision: typed decisions with calibrated probabilities on Gemma-4-12B, from text and images, one pass | |
| decider-12b-vision answers typed decisions with a probability for every option. It reads each question in one forward pass and | |
| generates no text. The state may include images. | |
| - **Input:** a state and one or more typed questions (Choice, Noul yes/no, Score), each with an explicit option list, in the | |
| System One wire format (`POST /v1/systemone`). Images are sent as Jev-style `assets` and referenced from the state. | |
| - **Output:** a probability distribution over the options of every question. | |
| - **Weights:** [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) with a LoRA fine-tune (rank 32, attention | |
| and MLP) merged into the weights, bf16, about 23 GB. The vision tower is the base model's, unchanged. | |
| - **Code:** `decider.serve_vllm` in [decider-ai](https://github.com/Mapika/decider) 1.10.0 for text and images; | |
| `decider.serve` and `decider.infer.Decider` for text. | |
| ## Which 12B to use | |
| | | [decider-12b](https://huggingface.co/Mapika/decider-12b) v2 | decider-12b-vision | | |
| |---|---|---| | |
| | Decision Index 0.3 sample (4,500 requests) | 51.74 | **56.74** | | |
| | Image questions, mean of 11 sets | 0.796 | **0.870** | | |
| | JevBench public items, hard tier | 79 / 111 | **82 / 111** | | |
| | Fresh held-out yes/no set, scored the JevBench way | **68.1** | 65.0 | | |
| | Yes/no temperature in the config | 0.05 (JevBench's 0.2-0.8 rule) | fitted, calibrated | | |
| decider-12b v2 stays the model we submit to JevBench: its yes/no answers score higher under JevBench's rule, and its config is | |
| set for that board. decider-12b-vision is better on the Decision Index task mix, reads images, and gives calibrated yes/no | |
| probabilities. | |
| ## Results | |
| All numbers measured by us; none were verified by the benchmark maintainers. | |
| | | stock Gemma-4-12B-it, our readout | decider-12b v2 | decider-12b-vision | | |
| |---|---|---|---| | |
| | Decision Index 0.3, 4,500-request sample | | 51.74 | **56.74** | | |
| | Image questions, mean of 11 sets | 0.781 | 0.796 | **0.870** | | |
| | Visual7W (not trained) | 0.880 | 0.880 | 0.883 | | |
| | VSR (not trained) | 0.750 | 0.787 | 0.803 | | |
| - The index sample is the 4,500-request sample of the 0.3 public suite; the full public suite was not run on this model. Through | |
| the released server the sample gives 56.63. | |
| - Image questions: the eval splits of 11 sets from [the_cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron), | |
| 300 items each. Visual7W and VSR were never trained on. Through the released server with the image sent as an asset: | |
| Visual7W 0.880, VSR 0.800. | |
| - Our own held-out sets against decider-12b v2: CLINC150 0.967 (0.903), iSarcasmEval 0.813 (0.707), VAST 0.793 (0.710), ACOS | |
| 0.987 / 0.853 (0.943 / 0.803), POP909-CL chords 0.690 (0.103), smart-home requests 0.998 (0.929). | |
| - The temperatures were fitted on our own rows only; no Decision Index row, JevBench item or image evaluation item was used to | |
| fit or choose them. | |
| ## Usage | |
| ```bash | |
| python -m venv decider-vllm && . decider-vllm/bin/activate | |
| pip install vllm==0.29.0 fastapi "uvicorn[standard]" jinja2 huggingface_hub pillow | |
| pip install --no-deps "decider-ai>=1.10.0" | |
| DECIDER_MODEL=Mapika/decider-12b-vision uvicorn decider.serve_vllm:app --host 127.0.0.1 --port 8000 | |
| ``` | |
| A request with an image (the asset's key is the reference; the state names it): | |
| ```python | |
| import base64, httpx | |
| img = base64.b64encode(open("receipt.png", "rb").read()).decode() | |
| r = httpx.post("http://127.0.0.1:8000/v1/systemone", timeout=60, json={ | |
| "state": {"receipt": "receipt-1", "policy": "Meals over 60 EUR need a manager's approval."}, | |
| "assets": {"receipt-1": {"type": "image", "format": "png", "data_base64": img}}, | |
| "questions": {"approval": {"type": "noul", "instructions": "Does the meal on `receipt` need approval?", | |
| "criteria": {"true": "needs approval", "false": "no approval needed"}}}}) | |
| print(r.json()["answers"]["approval"]) | |
| ``` | |
| Text-only use works the same without `assets`, and also through `decider.serve` or `Decider("Mapika/decider-12b-vision")` | |
| (text only; those start without image input and say so). | |
| ## How a request is answered | |
| 1. **First reading.** Every question is rendered in the model's chat template with thinking off, and the log-probabilities of | |
| the option letters are read at the answer slot. A softmax at T1(n) = 2.078 − 0.036 ln n (n = number of options) gives the | |
| probabilities. Each image in the state is one placeholder, which vLLM expands to the image's soft tokens (at most 280). | |
| 2. **Second reading, only when unsure** (`decider.serve_vllm`): a question whose top probability is below 0.7 is read again with | |
| its options reversed; the two log-softmaxes are averaged and a softmax at T2(n) = 2.025 − 0.042 ln n gives the final | |
| probabilities. | |
| ## Training | |
| Two LoRA stages (rank 32, alpha 64) on google/gemma-4-12B-it, then merged: | |
| 1. 955 steps at learning rate 5e-5 (28.0M tokens, 2.5 hours on one B300) on 18,000 text rows from the decider-31b training | |
| mixture and 18,129 image rows. | |
| 2. 580 steps at 2e-5 (16.9M tokens) on the state-tracking mixture of decider-12b v2 (10,000 rows), plus 3,000 image rows and | |
| 4,000 text rows of stage 1 replayed. | |
| - **Image rows:** train splits of seven the_cauldron subsets: AI2D, A-OKVQA, Hateful Memes, IconQA, RAVEN, ScienceQA, TQA. | |
| Yes/no questions are trained as Noul questions, the others as Choice questions. | |
| - **Text rows:** the decider-31b mixture (train splits of CLINC150, iSarcasmEval, VAST, ACOS, POP909-CL, GSM8K and NLI4CT, | |
| decontaminated against every Decision Index 0.3 suite row; generated smart-home requests; generated state-tracking decisions | |
| and decision families; document questions written by Qwen3.6-27B; labelled rows from public datasets). | |
| - Not used: Decision Index test rows, JevBench or ImageJevBench items. | |
| ## Limitations | |
| - **One pass only:** multi-step reasoning tasks are not solved by reading harder. | |
| - **Images:** the gain is on the kinds of image questions trained; on new image question types it reads about as well as the | |
| stock model. Box answers and image references in question text are not supported. Not measured on ImageJevBench. | |
| - **Latency** was not measured for this model. | |
| - **JevBench:** use decider-12b v2 for that board (see above). | |
| ## License | |
| The weights are released under Apache 2.0, as the base model. Training sources and their terms: | |
| - Image questions, from the train splits of [the_cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) subsets | |
| (the_cauldron licenses only its own prompts, CC BY 4.0; each source keeps its own terms): | |
| - AI2D (Kembhavi et al., 2016): CC BY-SA 4.0. | |
| - A-OKVQA (Schwenk et al., 2022): Apache 2.0 (repository); images from COCO. | |
| - IconQA (Lu et al., 2021): CC BY-NC-SA 4.0, non-commercial. | |
| - ScienceQA (Lu et al., 2022): CC BY-NC-SA 4.0, non-commercial research use. | |
| - TQA (Kembhavi et al., 2017): CC BY-NC 3.0, non-commercial. | |
| - RAVEN (Zhang et al., 2019): no data licence stated (the code is GPL-3.0). | |
| - Hateful Memes (Kiela et al., 2020): Facebook's Hateful Memes dataset licence. It permits use of the data to research and | |
| develop models designed to detect manipulated media, and states that models derived from it may be used for academic and | |
| commercial purposes. This is a research project; check the licence for your own use. | |
| - CLINC150: CC BY 3.0 (Larson et al., 2019). Credit: "An Evaluation Dataset for Intent Classification and Out-of-Scope | |
| Prediction", clinc/oos-eval. | |
| - iSarcasmEval: MIT (Abu Farha et al., 2022). The texts are tweets. | |
| - POP909-CL: MIT; built on POP909 (Wang et al., 2020), MIT. | |
| - VAST (Allaway and McKeown, 2020) and ACOS (Cai et al., 2021): no licence stated by the authors. | |
| - GSM8K (Cobbe et al., 2021): MIT. | |
| - NLI4CT (SemEval-2024 Task 2, Jullien et al.): no licence file in the task repository; released for the shared task. | |
| - RACE: non-commercial research use. It is a small part of the replayed data (250 rows). | |
| - ARC, SNLI, LogiQA 2.0 (copy dependent): CC BY-SA 4.0; BoolQ: CC BY-SA 3.0; BANKING77, MedQA: CC BY 4.0; MMLU, | |
| CommonsenseQA: MIT; MNLI: mixed per genre. | |