Instructions to use h2oai/h2o-lightning-12b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2o-lightning-12b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="h2oai/h2o-lightning-12b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("h2oai/h2o-lightning-12b")
model = AutoModelForMultimodalLM.from_pretrained("h2oai/h2o-lightning-12b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2o-lightning-12b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2o-lightning-12b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-12b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/h2oai/h2o-lightning-12b

SGLang

How to use h2oai/h2o-lightning-12b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2o-lightning-12b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-12b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2o-lightning-12b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-12b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use h2oai/h2o-lightning-12b with Docker Model Runner:
```
docker model run hf.co/h2oai/h2o-lightning-12b
```

Accept the H2O Model Non-Commercial Licence to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is released under the H2O Model Non-Commercial Licence v1.1 (see the LICENSE file): non-commercial, non-production use only.

H2O-Lightning-12B

H2O-Lightning-12B is a 12B-parameter decision model from H2O.ai, built on google/gemma-4-12B-it. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation), and about images that come with the record. It returns a probability for every option:

choice: picks one of a set of named options.
yes/no (noul): gives the probability that a statement is true.
score: gives an ordinal level on a stated scale, with its probability distribution.

This card describes v1.0 (main; pin --revision v1.0).

This release

This card describes v1.0 (main; pin --revision v1.0). Compared with the base model, google/gemma-4-12B-it, on the Decision Index 0.3 public suites, both read on one NVIDIA RTX PRO 6000 (96 GB) per server, this release with the decision-index profile (bf16 weights, two-way reading) and the base model through the same shim, prompt and kit, read one-way (it was not trained for the two-way reading):

Decision Index 0.3 public	gemma-4-12B-it	this release	difference
text index	50.18	63.92	+13.74 [+13.09, +14.32]
vision index	65.37	67.97	+2.60 [+1.25, +3.97]

The text difference is a bootstrap over the kit's case groups (10,000 draws); the vision difference is paired over the 12,210 public vision rows, resampled within each benchmark (10,000 resamples). The text index is over the kit's rows minus 4,999 rows whose items overlap this model's training data; on all rows it is 63.93 against 50.36 (+13.57 [+12.96, +14.19]). The probability constants were fitted on these weights (see Calibration).

Below the base model. On text, 5 of the 43 benchmarks score under the base model: PhishNChips -3.2, BRIGHT -1.6, API-Bank -1.4, HLE -1.0, RouterBench -0.2 (points of each benchmark's own metric). PhishNChips is the one whose shortfall is clear of noise (verdict accuracy 80.3 % against 83.5 %, a difference of -3.2 points [-4.8, -1.7]); see Limitations. On vision, 4 of the 11 benchmarks score under it: Winoground -4.2, RealWorldQA -4.2, BLINK -2.1, KIE (CORD+FUNSD) -0.7 (points).

How to run it (recommended setup first):

Recommended: our vLLM build, two-way reading. Our per-request attention build of vLLM 0.30.0, with the shim (lightning_12b_shim.py) in front, reads each request two-way (one-way for a long or heavy request). Three profiles:
- decision-index, the Decision Index runbook (bash h2o-lightning-12b/run_decision_index.sh): every record read whole, a text choice or yes/no answer under 0.90 read a second time (the two reads averaged), an unsure image question read again at 1120 soft tokens.
- two-way, for JevBench and for API serving with one read per question (bash h2o-lightning-12b/run_default.sh, or serve.sh --profile two-way).
- decision-index-max, opt-in: the same re-read below 0.95.
Default configuration (AA mode): bash h2o-lightning-12b/serve.sh with no settings, on our build: the Decision Index setup read two-way, plus the "I don't know" rule (see Default configuration (AA mode)).
Fallback: stock vLLM 0.30.0, one-way reading, for a server without our build: READING=one-way bash h2o-lightning-12b/serve.sh (see the fallback). Measured on this release: text index 63.73, vision 67.16.
Modes: each request is answered in one of three modes (jevbench, decision_index, aa), named in the request or set by the server's default; see Modes.

Two-way against one-way. With the Decision Index settings, the two-way reading scores +0.19 [-0.09, +0.46] on text and +0.81 [-0.07, +1.67] on vision over the one-way fallback (the same files and settings; only the reading differs).

JevBench (in-house, the 231 public items)

Measure (this release, `bash h2o-lightning-12b/run_default.sh`, one RTX PRO 6000)	Value
JevBench, 231 public items	207 / 231 (0 errors)
latency, the 231 request bodies sent once each, one at a time (client wall time; warm-up requests not counted)	median 40.1 ms, mean 81.3 ms, 80th percentile 90.0 ms
input tokens per decision	649.7 (mean `usage.input_tokens`)

Our own client sent each item once, one at a time; the board computes Intelligence, Calibration and Cost on its own items, so no board score is quoted here. This latency is for these 231 JevBench requests in jevbench mode (run_default.sh's default).

Calibration (constants fitted on these weights)

Every temperature and the yes/no floor were fitted on these weights from Decision Index 0.3 public per-row reads (halves split by a hash of each case, each value accepted only when both halves agree, the calibration error lower on both halves), and confirmed on a fresh set:

constant	value	basis
choice temperature, two-way reads	1.12	fitted (halves 1.12 / 1.12); confirmed on SuperGPQA (1,990 clean questions): calibration error 0.166 -> 0.035
yes/no temperature, two-way reads	0.93	fitted (halves 0.93 / 0.94); in-sample
choice temperature, one-way reads	1.12	fitted; confirmed on SuperGPQA: calibration error 0.194 -> 0.051
yes/no temperature, one-way reads	1.08	fitted; in-sample
score temperature	0.7	not fitted (no Decision Index score rows)
image temperature, decision-index profiles (both readings)	0.91	fitted on the two-way first read; confirmed on MMStar (1,493 questions): calibration error 0.209 -> 0.078
image temperature, other profiles (`serve_config.json` image block)	0.92 one-way, 0.91 two-way	fitted; one-way confirmed on MMStar: calibration error 0.219 -> 0.088
yes/no floor (both readings)	off (0)	fitted: no floor is better on both halves
"I don't know" cutoff (`aa` mode)	0.65	fitted on a real-"I don't know" read of these weights; see Modes
box confidence map (`serve_config.json` `box`)	none (the raw readout)	checked on RefCOCOg validation (600 questions, board images excluded): calibration error 0.050; a fitted map did worse held out on one half, so none is applied

The confirmation sets were decontaminated against this model's training data first (SuperGPQA: 1,990 clean questions of 2,000; MMStar: 1,493 clean items).

The image re-read of the decision-index profiles is gated on the tempered top probability (under 0.8), so the image temperature also sets how often it fires: at 0.91 it re-read 4,104 of the 12,210 public vision rows (33.6 %). A fit on the 1120 re-reads alone gives 1.02; it is not used, because those rows are the ones the first-read temperature selects.

Modes

Each request is answered in one of three modes. A request names its mode in "mode" (or kwargs.mode); a request without one gets the server's default: decision_index under run_decision_index.sh (the decision-index profile), jevbench under run_default.sh (the two-way profile), and aa for serve.sh with no profile.

mode	`"mode"`	what it does
JevBench	`"jevbench"`	the server's own settings (under `run_default.sh`: one read per question, a record over 126,976 tokens cut down by BM25)
Decision Index	`"decision_index"`	the `decision-index` profile's reading: every record read whole up to the context, the conditional text re-read under 0.90, the image re-read at 1120 under 0.8
AA (`"auto"` is the same)	`"aa"` or `"auto"`	Decision Index, plus the "I don't know" rule: a choice question with an explicit "I don't know" option answers it when its top real answer, renormalised without that option, is under 0.65

The "I don't know" cutoff 0.65 was fitted on these weights from a real-"I don't know" read (4,398 scored questions; the lowest cutoff within noise of the best, 0.75, on our questions, in the shipped aa reading); the AA index gain over always answering is about +8.3. On easier families (MMLU, MMLU-Pro) 0.65 gains on 0.80 (+0.66 index); on harder ones (GPQA, HLE, about 26 % accuracy) it loses (-1.01). If your questions are much harder than ours, a higher cutoff (0.90) would score better. Every response says its mode in effort_used.mode; /health lists the modes and their settings.

Default configuration (AA mode)

bash h2o-lightning-12b/serve.sh with no settings starts the AA setup: the Decision Index setup read two-way on our vLLM build, plus the "I don't know" rule. A request that names no mode is answered in aa mode.

What it reads. The decision_index mode's settings: every record read whole (up to 126,976 tokens, or the context minus 4,096 on a GPU with a smaller context; a longer record is refused with HTTP 422, never cut), a text choice or yes/no answer under 0.90 read a second time and the two reads averaged, an unsure image question read again at 1120 soft tokens. Each request is read two-way, and one-way, read whole, when its record is longer than the two-way window or its two-way prompts add up to more than 4,096 tokens, or it has a Box question (see Two-way reading).
The "I don't know" rule. It applies only to a choice question that offers an "I don't know"-type option. The answer is read as in decision_index mode; then the real options (every option except the "I don't know" one) are renormalised to add up to 1. If the most likely real option is under 0.65, the answer is the "I don't know" option (reported at 0.65); otherwise the answer is unchanged. Every other question is answered exactly as in decision_index mode. Each answer to such a question reports the rule in idk (the option, the cutoff, the top real probability and abstained).
Which options count. An option whose text (or name), lower-cased, with a typographic apostrophe read as ' and end punctuation dropped, is one of serve_config.json idk.labels: "i don't know", "i do not know", "don't know", "do not know", "dont know", "idk", "not sure", "i'm not sure", "i am not sure", "unsure", "no idea". A question with two such options gets no rule.
Start it. One command, on one GPU of a measured class (the per-GPU table), in the Python environment that holds our vLLM build (run_decision_index.sh installs it into ./di-venv on its first run):
```
bash h2o-lightning-12b/serve.sh
```
Without our build, or on a GPU without measured sizes, it still starts, but reads every request one-way (as READING=one-way does) and says so loudly: a warning at start naming the reason and both fixes (install the build, or set MAX_LEN and TWO_WAY_WINDOW for that GPU; or start READING=one-way bash h2o-lightning-12b/serve.sh to choose one-way and silence the warning), and /health shows "reading": "one-way (fallback: <reason>)" with the same words under "warnings". READING=two-way asks for two-way explicitly and stops instead when it cannot be served. If the engine is our build but its two-way start-up check does not pass, the server reads every request one-way and says so (see Two-way start-up check).
Turn it off. Per request: "mode": "decision_index" (the same reads without the rule) or "mode": "jevbench". Per server: bash h2o-lightning-12b/serve.sh --profile decision-index (default mode decision_index; the decision-index profile, sized for one RTX PRO 6000) or --profile jevbench (default mode jevbench).
Check it. curl -s http://127.0.0.1:8741/health shows "default_mode": "aa", "reading": "two-way (verified)" ("one-way (fallback: <reason>)" when it serves one-way instead, as above), and under "modes", "aa", the cutoff "idk_cutoff": 0.65. Every response says its mode in effort_used.mode and its reading in effort_used.attention_mode.
Measured. bash h2o-lightning-12b/serve.sh with nothing set, on one RTX PRO 6000 from this release's files (the two-way start-up check verified): JevBench's 231 public items, each sent once, one at a time, in the server's default aa mode: 208 / 231 (0 errors); latency median 46.7 ms, mean 102.0 ms, 80th percentile 218.5 ms (client wall time; warm-up requests not counted); 649.7 input tokens per decision (mean usage.input_tokens).

Two-way start-up check

Before its first answer a server on our vLLM build checks that the engine really reads two-way: three short pairs of requests that share a prefix and differ after it are read flagged (two-way) and unflagged (one-way), and the unflagged reads once more. Two-way is served only when the flagged prefix moves with the later text (at least 0.1 in log-probability) while the unflagged prefix does not (at most 0.001) and the repeated unflagged read agrees with itself. If the check does not pass, the server does not refuse: it serves every request one-way, says so in /health (two_way_check.verified false, with the reason, and reading "one-way (fallback: <reason>)"), in a warning in its log every 10 minutes and in every answer (effort_used.attention_reason "two-way unavailable (start-up check failed)"), and answers an explicit two-way request with HTTP 400. The only refusal (HTTP 503) is a deterministic engine whose one-way reading sees later tokens. The benchmark runbooks stop instead of serving the fallback (their /health check expects two_way_check.verified). A fixed set of reference probes, recorded on these weights, is also read at start and reported in /health; it does not gate.

Recommended: how to run for the Decision Index

For Decision Index maintainers: run it this way. The engine sizes are for one NVIDIA RTX PRO 6000 (96 GB), the GPU every number on this card was measured on. One shipped command installs the pinned engine, serves the model with the settings the expected numbers below were measured with, checks the running server against every value of the profile, and only then says it is ready. There is nothing to choose and no setting to type: run the one command in step 2.

What it serves: our vLLM build, reading each request two-way (the model's trained two-way reading, full-attention layers looking both ways) and switches a request to one-way, read whole, when its record is longer than the two-way window or its two-way prefill would be slow. No record is ever cut, selected or summarised.

Machine. One NVIDIA RTX PRO 6000 (96 GB), NVIDIA driver 580 or newer (vLLM 0.30.0 installs PyTorch's CUDA 13.0 build), Linux, Python 3.10 to 3.12 with its venv module, a C compiler, curl and git. Disk: about 24 GB for the download and about 12 GB for the engine's environment (and about 17 GB to rebuild the vision suite). On a minimal image such as pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime:

apt-get update && apt-get install -y gcc g++ curl git cabextract && pip install uv

1. Download this release. The repository is gated: accept the licence on the model page, then log in.

pip install -U huggingface_hub && hf auth login
hf download h2oai/h2o-lightning-12b --revision v1.0 --local-dir h2o-lightning-12b

2. Start the server with the one shipped command:

bash h2o-lightning-12b/run_decision_index.sh

What it installs. On the first run only, vLLM 0.30.0 (with the measured PyTorch 2.13.0, transformers 5.20.0 and torchcodec 0.17.0) into ./di-venv (beside where you run it), with our per-request attention build laid over it: the changed files of h2oai/vllm at commit b05cc29ab813895947954c268c4ec41ca0da1306, each checked against engine/vllm_per_request_attention.SHA256SUMS. Every later run checks the version and every file again, and refuses a changed engine. No package is installed outside ./di-venv; vLLM's compile caches go under ~/.cache/vllm/h2o-lightning-12b/, one folder per engine build, reading mode, sizes, image setting and weight format, so no setting reuses another's compiled graphs (VLLM_CACHE_ROOT chooses another folder; both runbooks pass it through).

It does four things, and stops with REFUSE: and the reason on any mismatch:

a. checks the NVIDIA driver (580 or newer), Python, a C compiler, curl and git, and that the shipped files are this release's (engine/package.SHA256SUMS); b. installs or checks the engine, as above; c. runs serve.sh --profile decision-index in that environment: the GPU's memory checked against the profile's GPU, vLLM started with the arguments the shim prints, the shim started with --profile decision-index. The profile's settings are below. Every setting in this table is a default that --profile decision-index (and run_decision_index.sh, which uses it) applies by itself: you set none of them when you start with the Decision Index option, and the shim refuses any of them from the environment.

setting	value
engine	our vLLM build, reading each request two-way or one-way
weights	bf16 (the checkpoint as shipped; no quantization)
context (`MAX_LEN`)	131,072 tokens, every Decision Index record read whole
two-way	a request whose questions' two-way prompts add up to at most the latency cap below, and whose record fits the two-way window (8,192 tokens; `TWO_WAY_WINDOW`)
one-way, read whole	any other request: one whose summed two-way prompt is over 4,096 tokens (the latency cap)
readout orders	one read per question
conditional re-read	a text choice or yes/no answer whose top probability is under 0.90 is read a second time, with the request written out twice, and the two reads are averaged; only for requests of up to 2,000 tokens
images	one read at Gemma-4's 560-token image budget; a choice or score image question whose top probability is under 0.8 is read again at 1120 and the two reads averaged
Box questions	on (as in normal serving; no setting): one guided generation per box question, read one-way
record (`state`)	read whole up to 126,976 tokens; a record that does not fit is refused with HTTP 422, never cut
temperatures and yes/no floor	one-way choice 1.12, score 0.7, yes/no 1.08; two-way 1.12, 0.7, 0.93; images 0.91 in both readings; floor off
default mode	`decision_index` (a request may ask for `aa` or `jevbench`)

For reference only (the profile sets them; do not export them): the re-read row is SHIM_REREAD_BELOW=0.9, SHIM_REREAD_COMBINE=avg, SHIM_REREAD_MAX_TOKENS=2000, SHIM_REREAD_LAYOUT=header; the image row is SHIM_IMAGE_REREAD_BUDGET=1120, SHIM_IMAGE_REREAD_BELOW=0.8, SHIM_IMAGE_ORDERS=1. See Conditional re-read and Image re-read.

d. waits for the server and compares its /health with the profile, field by field (lightning_12b_shim.py --profile decision-index --check-health): "ok": true; profile.name "decision-index"; serving.attention_mode "auto"; the two-way start-up check verified (two_way_check.verified); auto.engine "one", auto.max_total_tokens the cap and auto.two_way_window the window; max_state_tokens 126976 and selector "none"; the temperatures of both readings (one-way choice 1.12, score 0.7, yes/no 1.08; two-way 1.12, 0.7, 0.93; images 0.91 in both); both yes/no floors off (0); readout orders "off" (one read); the image re-read {"budget": 1120, "below": 0.8} in both readings, with image orders 1.

Before it reports ready, serve.sh sends one warm-up request down every path this configuration serves (warmup.py: two-way, several questions at once, a request over the latency cap, a record over the two-way window, a long record, an image at the 560 budget and at the 1120 re-read budget), so no real request pays a first-use compile; until then /health answers 503 with "warming": true. When it prints DECISION INDEX SERVER READY on http://127.0.0.1:8741, the server is the measured configuration. Every answer states which reading served it (effort_used.attention_mode and effort_used.attention_reason).

3. Run the Decision Index kit (edition 0.3) against the server, from a second shell (the server keeps the first) and in its own environment:

uv venv --python 3.12 kit && VIRTUAL_ENV=kit uv pip install "decision-index[vision] @ git+https://github.com/apolinario/decision-index@f8552a1" pillow==13.0.0 numpy
kit/bin/decision-index suite download --edition 0.3 --dir suite-0.3
kit/bin/decision-index run --edition 0.3 --suite-dir suite-0.3 --engine http --option base_url=http://127.0.0.1:8741 --option model=h2oai/h2o-lightning-12b --out runs/text
kit/bin/decision-index score --edition 0.3 --suite-dir suite-0.3 --results runs/text/results.jsonl --engine http --out runs/text/score
kit/bin/decision-index vision run --dir <vision suite dir> --engine decide --option base_url=http://127.0.0.1:8741 --option path=/v1/systemone --option protocol=decisions --option model=h2oai/h2o-lightning-12b --option max_pixels=1638400 --workers 16 --out runs/vision
kit/bin/decision-index vision score --dir <vision suite dir> --results runs/vision/results.jsonl --out runs/vision/score

<vision suite dir> is WORK/vision, written by the rebuild in the last item below, or the pinned vision suite you already hold (decision-index vision verify must pass on it).
One text run in a single process takes days on one GPU. The measured text run was split, as a speed-up only: the kit's own row list (rows unchanged) into the 16 files of the shipped di_tools/di_text_sharded.sh, read by three servers of this package (one per GPU, each file resumed by the kit from its own results), the untouched results concatenated and scored once by decision-index score. di_tools/di_text_sharded.sh (16 files round-robin against one server) replaces the text run and score lines above (it writes runs/text/results.jsonl and scores it into runs/text/score). A different split batches requests differently on the engine, so individual answers can move as between two compiles. It needs decision-index on the PATH, e.g. the kit environment activated:
```
. kit/bin/activate
bash h2o-lightning-12b/di_tools/di_text_sharded.sh suite-0.3 http://127.0.0.1:8741 h2oai/h2o-lightning-12b runs/text 16
```
Build the vision suite with the kit's own builder, giving it Microsoft's core-fonts Arial Bold: the kit draws Mind2Web's box labels in Arial Bold, and only that font reproduces the pinned images. The file is arialbd.ttf from arial32.exe (the core-fonts package Ubuntu's ttf-mscorefonts-installer downloads, from https://downloads.sourceforge.net/corefonts/arial32.exe; sha256 of arial32.exe 85297a4d146e9c87ac6f74822734bdee5f4b2a722d7eaa584b7f2cbf76f478f6, of arialbd.ttf 4044aa6b5bebbc36980206b45b0aaaaa5681552a48bcadb41746d5d1d71fd7b4):
```
curl -L -o arial32.exe https://downloads.sourceforge.net/corefonts/arial32.exe   # check its sha256 against the one above
cabextract -L -F '*.ttf' arial32.exe        # gives arialbd.ttf; check its sha256 against the one above
kit/bin/decision-index suite rebuild --vision --work WORK --font "$PWD/arialbd.ttf"
kit/bin/decision-index vision verify --dir WORK/vision
```
vision verify must pass on all 11 benchmarks; the vision command above then runs as written. Without that font the kit falls back to Pillow's default font, Mind2Web's image bytes differ from the pin, and the vision run stops before reading anything. For the measured suite only Mind2Web was rebuilt with this font, by the kit's own builder (the other ten sets come from an earlier build of the same kit), and the suite passed vision verify on all 11; a from-scratch suite rebuild --vision --font of all eleven was not run end to end.

How to tell it is right.

The command printed DECISION INDEX SERVER READY on http://127.0.0.1:8741: it says that only after the warm-up and after every /health value above matched the profile.
The kit's scores land at the expected numbers: Decision Index 0.3 public text index 63.93 (all rows, as decision-index score reports it) and vision 67.97, measured exactly this way (the decision-index profile, one RTX PRO 6000 per server) on this release's files. The engine reads many requests at once, and the engine's compile and kernel-tuning cache can differ between installs and shift probabilities slightly, occasionally flipping a near-tie answer, so a rerun on a fresh install can differ in the second decimal of the index.
In the kit's results, every response carries effort_used.attention_mode (bidirectional for a two-way read, causal for one-way), and an image response re-read at 1120 carries effort_used.image_reread.

What not to change.

No environment variables for the server: the command starts it with a clean environment (only the GPU selection, CUDA_VISIBLE_DEVICES, the library path, PATH, HOME, HF_HOME and the three sizes for another GPU pass through), and the shim refuses a setting of its profile from the environment.
No edits to the downloaded files: the command checks the files it serves with (serve.sh, the shim, its configs, the warm-up, the engine's checksums) against engine/package.SHA256SUMS and refuses a changed one.
Not stock vLLM: the one-way fallback below is a different configuration with its own numbers.
Not run_default.sh for the Decision Index: it is the two-way profile, which leaves out the 1120 image re-read and would cut a record over 126,976 tokens down with BM25 (no Decision Index record reaches that); this card's Decision Index numbers were not checked against it.
The kit commands as written in step 3, with max_pixels=1638400 for vision.

Latency. Not measured with this runbook on this release. The serial latency measured on this release is the JevBench one above (a different request set and mode). The board's latency rule (median, mean and 80th percentile each at most 1,000 ms on its own GPU) has not been checked for this runbook.

Troubleshooting.

Two copies on one machine: run_decision_index.sh always uses ports 8000 (vLLM) and 8741 (the shim), so run one copy per machine.
REFUSE: no C compiler or serve.sh needs a C compiler: vLLM compiles the model at start-up. Install gcc g++ (and curl git) as in Machine above.
REFUSE: ... is not the pinned per-request attention build: ./di-venv holds another vLLM. Remove the folder and run the command again; it reinstalls.
the two-way warm-up through the shim failed: the engine answered the two-way start-up check like a one-way engine, which is what stock vLLM does (it ignores the per-request flag). The cause is an engine that is not the pinned build; remove ./di-venv and run again.
Could not find nvcc at the first request: FlashInfer's sampler compiles itself on first use and needs the CUDA toolkit, which minimal images lack. The decisions never sample, so serve.sh turns that sampler off (VLLM_USE_FLASHINFER_SAMPLER=0); the error means it was turned back on in the environment.
CUDA out of memory at start: the engine takes 90% of the GPU's memory (GPU_UTIL 0.9); nvidia-smi shows what else holds memory on it.
the warm-up failed (above): a warm-up request (warmup.py, one per path) did not answer 200; the line above it names the path and the server's answer. A path that fails the warm-up would fail real requests the same way.
REFUSE: the server is not the decision-index profile's configuration: the lines under it name each /health field and the value the profile expects; a changed file in the download is the usual cause (download the release again).
This profile is for one NVIDIA RTX PRO 6000 Blackwell (96 GB) and this GPU has ... MiB: the GPU has less memory than the profile's minimum; see Another GPU below.
profile decision-index sets ... itself or ... would change the measured configuration: a SHIM_* variable is set in your environment; unset it (the profile sets the shim's settings itself).
the shipped configs are not the measured ones: serve_config.json or serve_config_two_way.json was changed; download the release again.
shim: ENGINE CHECK FAILED (and HTTP 503 on every request): vLLM was not started as the profile needs; the message names the command to start it with.

Other ways to start the same server (not needed for a measured run). run_decision_index.sh above is the measured run; the notes below are for maintainers who start the pieces themselves.

Where the settings live. Every Decision Index value is the shim's decision-index profile (lightning_12b_shim.py, PROFILES): the shim's settings, the vLLM engine's arguments and sizes, the vLLM version and the per-request attention commit. The script holds none of them; it installs the engine and starts vLLM and the shim with what the profile says. If you run the shim yourself, pass --profile decision-index: start vLLM (with our per-request attention build installed, as below) with the arguments the shim prints, then the shim with the profile. At start the shim checks the engine (the served model and its context, the chat rendering, the per-request attention build answering both readings) and refuses every request, naming the command to start vLLM with, until it matches; the weight format and memory settings are not visible to it, so start vLLM with the printed arguments as they are. bash h2o-lightning-12b/serve.sh --profile decision-index is the script's server without the install step (for an environment that has the engine).

. di-venv/bin/activate   # the engine's environment, as run_decision_index.sh installs it
python3 h2o-lightning-12b/lightning_12b_shim.py --profile decision-index --print-vllm-args \
  | VLLM_USE_FLASHINFER_SAMPLER=0 xargs -d '\n' vllm serve h2o-lightning-12b --host 127.0.0.1 --port 8000 &
python3 h2o-lightning-12b/lightning_12b_shim.py --profile decision-index --tokenizer h2o-lightning-12b/tokenizer.json

Another GPU. The profile's sizes are the RTX PRO 6000's; the command refuses a GPU with under 44,000 MiB of memory and has no other GPU logic. That minimum is not measured: no GPU smaller than 96 GB was run with this release. On another GPU set the three sizes yourself (MAX_LEN, TWO_WAY_WINDOW, GPU_UTIL); a record is then read whole only up to MAX_LEN - 4,096 tokens, and a longer one is refused with HTTP 422 and scored wrong; /health lists the sizes under profile.overrides. This release's numbers were measured at the profile's own sizes on one RTX PRO 6000.

Recommended: how to run for JevBench

Follow these steps literally; there is nothing to choose. The machine, the download and the engine are the Decision Index block's, and so are the settings, with two differences: a record over 126,976 tokens is cut down by BM25 selection instead of refused, and there is no conditional re-read and no 1120 image re-read.

Machine and download as in steps "Machine" and 1 of the Decision Index block. This runbook is the shim's two-way profile; it is also the default server for general use on our vLLM build.

Start the server with the one shipped command:

bash h2o-lightning-12b/run_default.sh

It does the same steps as run_decision_index.sh with the two-way profile (the GPU check, the pinned engine, every setting fixed, /health compared field by field) with BM25 selection over 126,976 tokens and no conditional re-read, and prints DEFAULT SERVER READY on http://127.0.0.1:8741 when the server is the measured configuration. Every setting in this table is a default that the two-way profile (and run_default.sh, which uses it) applies by itself: you set none of them.

setting	value
engine	our vLLM build, reading each request two-way or one-way, bf16 weights
reading	two-way by default; one-way, read whole, for a record over the two-way window or a request over the latency cap
readout orders	one read (`off`)
temperatures and yes/no floor	one-way choice 1.12, score 0.7, yes/no 1.08, images 0.92; two-way 1.12, 0.7, 0.93, images 0.91; floor off; default mode `jevbench`
record (`state`)	read whole up to 126,976 tokens; a longer record is cut down to fit by BM25 selection (`SHIM_SELECTOR=v2`), never refused (a record with no part matching the question keeps its head and tail around a `...` line instead, `effort_used.state_selection` `"no match; head+tail"`)
images and Box questions	on (as in normal serving; no setting): images at Gemma-4's 560-token budget, read once; one guided generation per box question, read one-way

Run JevBench with the JevBench harness against http://127.0.0.1:8741/v1/systemone (model name h2oai/h2o-lightning-12b), one request at a time: the board's speed rule reads serial latency. The numbers below came from our own client (not the board's harness) sending each of the 231 public items once, serially, to that endpoint.

Expected numbers: see JevBench above (this command).

Fallback: stock vLLM, one-way only

Without our vLLM build, stock vLLM 0.30.0 serves the model one-way (no two-way reading), with the same shim and the same files, started with READING=one-way (without it, serve.sh serves the same one-way server but warns that it is not the default two-way reading). With no other setting it is the AA setup read one-way: the Decision Index settings (every record read whole, the text re-read under 0.90, the image re-read at 1120 under 0.8) plus the "I don't know" rule, at the one-way temperatures.

Measured on this release (READING=one-way bash h2o-lightning-12b/serve.sh, nothing else set, one RTX PRO 6000, bf16): Decision Index 0.3 public text index 63.73 against the base model's 50.18 (+13.55 [+12.91, +14.14]; all rows 63.74), vision 67.16 against 65.37 (+1.79 [+0.48, +3.08]). The server ran in the environment of our build, reading one-way (the engine's two-way flag unused), which is stock vLLM's one-way reading; stock vLLM itself was not run for these numbers.

Hardware. One GPU with room for the bf16 weights (about 24 GB) and the context: measured on one RTX PRO 6000 (96 GB); smaller GPUs were not run with this release (set MAX_LEN and GPU_UTIL for one).

1. Install vLLM 0.30.0 in a fresh Python 3.12 virtual environment (the measured environment: vllm 0.30.0, torch 2.13.0, transformers 5.20.0, torchcodec 0.17.0, Python 3.12), and download this release. The machine also needs a C compiler and curl: vLLM compiles the model at start-up, and serve.sh stops with a message when either is missing. Minimal images such as pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime have neither (apt-get install -y gcc g++ curl). The repository is gated: accept the licence on the model page, then log in (hf auth login).

python3.12 -m venv .venv && . .venv/bin/activate
pip install "vllm==0.30.0" "torchcodec==0.17.0"
hf auth login
hf download h2oai/h2o-lightning-12b --revision v1.0 --local-dir h2o-lightning-12b

2. Serve with the shipped script and one setting:

READING=one-way bash h2o-lightning-12b/serve.sh

This starts vLLM on port 8000 and the shim on port 8741:

setting (stock vLLM, the measured fallback)	value
attention	one-way (stock vLLM generate runner)
weights	bf16 (the checkpoint as shipped; `serve_config.json` `"quantization": "none"`)
context (`MAX_LEN`)	131,072 tokens (the default)
default mode	`aa`: the Decision Index settings plus the "I don't know" rule (cutoff 0.65)
record (`state`)	read whole, up to 126,976 tokens; a record that does not fit is refused with HTTP 422, never cut
temperatures and yes/no floor	choice 1.12, score 0.7, yes/no 1.08; images 0.91 in `aa` and `decision_index` mode (0.92 in `jevbench` mode); floor off
Box questions	on wherever images are read (no setting); needs an engine with structured outputs

The Decision Index kit does not allow an engine to shorten its inputs; a refused record is scored as a wrong answer. The "I don't know" rule touches only questions that offer an "I don't know" option (2 of the kit's text questions); "mode": "decision_index" in a request answers it without the rule.

Timeouts. The shim waits up to 180 s for vLLM's answer to a request (SHIM_TIMEOUT, in seconds), sends it once more on a fresh connection, and otherwise answers HTTP 502, which the kit scores as a wrong answer (and three failures in a row can stop a kit run).

3. Check that the server is ready and configured as above. curl -s http://127.0.0.1:8741/health must return "ok": true, "default_mode": "aa" and "reading": "one-way (READING=one-way)", and under "modes", "aa", the cutoff "idk_cutoff": 0.65.

Then run the kit as in step 3 of the Decision Index section.

API serving on stock vLLM: the jevbench mode (3,000-token selection)

For general API serving on stock vLLM, not for either benchmark: READING=one-way bash h2o-lightning-12b/serve.sh --profile jevbench with no other settings. It is the one-way fallback above, except that a request with no mode reads once (no re-reads, no "I don't know" rule) and long records are selected. A state of up to 3,000 tokens is sent whole. From a longer one, the shim keeps the parts most relevant to the question: whole paragraphs, lines or fields, ranked by BM25 against the question and its options, with their headings, up to 3,000 tokens. The question and its options are never cut. When no part of a long record matches the question, or the record cannot be split into fitting parts, it is cut to its head and tail around a ... line and answered (effort_used.state_selection "no match; head+tail"); selection never refuses. This keeps every request short and its latency even.

Not measured on this release: how much the 3,000-token selection costs against reading long records whole.

Inference options

For the Decision Index, use only the runbook block above; nothing in this section changes it. The same holds for JevBench and its block. This section lists every option the model has, so that other uses (your own API serving, other benchmarks) can choose; for each it says whether it is on in the two benchmark runbooks (where the profile sets it, and you set nothing) and what it is for elsewhere. "Use it for" never means "add it to a benchmark runbook".

Measurements of the options. On this release only the runbooks' own numbers above, the fallback and the two-way against one-way line are measured. Each option below says what it does and where it is on.

Two-way reading

What it does: the full-attention layers read the prompt both ways (the reading the model was trained for), per request, on our vLLM build.
Decision Index runbook: on. JevBench runbook: on. Set by the profiles.
Elsewhere: needs our vLLM build (plain serve.sh reads two-way by default, or serve.sh --profile two-way); stock vLLM reads one-way only (READING=one-way). Use it for: any serving on our build. Measured: +0.19 [-0.09, +0.46] text index and +0.81 [-0.07, +1.67] vision over the one-way fallback with the Decision Index settings.

Routing long or heavy requests one-way

What it does: a record longer than the two-way window, or a request whose questions add up to more than the latency cap of two-way prompt tokens, is read one-way, whole.
Decision Index runbook: on (cap 4,096 summed two-way prompt tokens; window 8,192). JevBench runbook: on (the same). Set by the profiles. A request is read two-way only when its questions' prompts add up to at most the cap. In the measured Decision Index text read, 13,727 of the 140,620 rows were read one-way this way.
Elsewhere: SHIM_AUTO_MAX_TOTAL_TOKENS sets the cap. Use it for: keeping two-way latency even.

Two-order reading (a second read of an unsure choice question)

What it does: a choice question whose first read is unsure is read again with its options reversed, and the two reads are averaged.
Decision Index runbook: off. JevBench runbook: off. (The Decision Index runbook's conditional re-read below is a different second read.)
Elsewhere: off by default, and this release ships no fitted cutoff for it (serve_config.json readout.cutoff is empty); SHIM_READOUT_ORDERS=auto with SHIM_READOUT_CUTOFF_CHOICE turns it on. Not measured on this release.

Third and fourth reads

What it does: a choice question still unsure after two reads is read in rotated option orders (or once in the other reading) and all reads are averaged.
Decision Index runbook: off. JevBench runbook: off.
Elsewhere: no switch; not offered.

Conditional re-read (a second read of an unsure text answer)

What it does: a text choice or yes/no answer whose top probability is under a threshold is read a second time, with the request written out twice, and the two reads are averaged. Only requests of up to 2,000 tokens are re-read. Score questions are never re-read; images get the image re-read instead.
Decision Index runbook: on, below 0.90 (the decision-index profile, also accepted as balanced); in the measured text read it re-read 87,837 of the 140,620 rows. JevBench runbook: off (the two-way profile reads once).
Elsewhere: serve.sh --profile decision-index-max re-reads below 0.95: more re-reads; not measured on this release. Use the re-read for: accuracy on unsure text answers when a higher median latency is acceptable.
Attribution: The re-read writes the request in FlexDM v1.0's header layout (FlexDM, MIT; the layout is Will Cole's, MIT), with decider's chat-layout parts (Apache-2.0) and shisa-ai/shisa-de-2's read-twice; the notices are in NOTICE.

Image re-read at a larger image budget

What it does: an image choice or score question whose top probability is under 0.8 is read again at Gemma-4's 1120-token image budget (instead of 560) and the two reads are averaged.
Decision Index runbook: on (1120, under 0.8). JevBench runbook: off. Set by the profiles.
Elsewhere: off by default (SHIM_IMAGE_REREAD_BUDGET=1120 SHIM_IMAGE_REREAD_BELOW=0.8 turns it on). Use it for: accuracy on images when latency matters less; it reads the image a second time, which is why the JevBench runbook (whose speed score reads the 95th percentile) leaves it off.

Per-type temperatures and the yes/no floor

What it does: each answer's probabilities are tempered by question type (fitted on these weights from Decision Index 0.3 public per-row reads; see Calibration). The yes/no commit floor (a yes/no answer reported at no less than the floor on its chosen side) is off in this release: the fit chose no floor.
Decision Index runbook: on. JevBench runbook: on. Stock vLLM one-way: on. The shipped values: one-way choice 1.12, score 0.7, yes/no 1.08, images 0.92; two-way 1.12, 0.7, 0.93, images 0.91; floor off. The decision-index profiles, and the aa and decision_index modes, read images at 0.91 in both readings.
Elsewhere: on as shipped; SHIM_TEMPERATURE*, SHIM_IMAGE_TEMPERATURE and SHIM_NOUL_FLOOR (a value in (0.5, 1) turns a floor on) change them. See Other settings.

Long records: read whole, or cut down by BM25

What it does: a record over the token budget is either refused (read whole, HTTP 422) or cut down to its most relevant parts by BM25 selection (with no match, its head and tail are kept; never refused).
Decision Index runbook: read whole up to 126,976 tokens, refused above (the kit forbids shortening inputs, and its longest record fits). JevBench runbook: whole up to 126,976 tokens, BM25 above.
Stock vLLM one-way, the fallback command: read whole up to 126,976 tokens. Plain serve.sh (aa mode): read whole, as the Decision Index runbook. The jevbench mode on plain serve.sh: BM25 down to 3,000 tokens. Use it for: general API serving, where short prompts keep latency even.

Weights: bf16 or FP8

What it does: the bf16 checkpoint as shipped, or vLLM's online FP8 of it.
Decision Index runbook: bf16. JevBench runbook: bf16. Stock vLLM one-way: bf16 (serve_config.json "quantization": "none").
Elsewhere: QUANT=fp8 bash serve.sh serves FP8 on an FP8-capable GPU (not available with --profile, which fixes the engine). Every number on this card is bf16. See bf16 or FP8 weights.

Bounding-box questions

What it does: a box question answers with a bounding box in an image.
Decision Index runbook: on. JevBench runbook: on. Everywhere else: on. There is no setting to turn it off: a box question on an image deployment is answered by one guided (JSON-schema) generation. An engine without structured outputs refuses it with HTTP 422; requests without a box question are unaffected. Not measured on the Decision Index. See Box questions.

Structured-output backend: llguidance (opt-in)

What it does: GRAMMAR_BACKEND=guidance bash serve.sh starts vLLM with llguidance (no free whitespace) instead of vLLM's default xgrammar, for the letter mask of every read of up to 26 options and the Box generation.
Decision Index runbook: off. JevBench runbook: off. Not available with --profile (a profile fixes the engine).
Elsewhere: off by default. Every number on this card was read with xgrammar.

FlashAttention 4 on Hopper (opt-in)

What it does: ATTENTION_BACKEND=fa4 bash serve.sh runs the two-way reads on FlashAttention 4 instead of the Triton backend, on Hopper GPUs (H100, H200) only.
Decision Index runbook: off. JevBench runbook: off. Not available with --profile. Not measured on this release.

Read timeout and the start-up checks

What it does: a vLLM answer slower than 180 s is sent once more, then answered HTTP 502; before its first answer the shim checks that the engine reads both ways (see Two-way start-up check).
Decision Index runbook: on. JevBench runbook: on. Stock vLLM one-way: the timeout is on (SHIM_TIMEOUT).
Start-up: when the shim refuses a running engine for good (the chat rendering, the context, the readouts, a deterministic engine whose one-way reading sees later tokens), serve.sh stops at once and prints the shim's reason.

Option details

The measurements behind the options above. None of them changes a benchmark runbook.

Two-way reading (on in both runbooks; needs our vLLM build)

The model was trained to read with its full-attention layers looking both ways (two-way, or bidirectional, attention) as well as one-way. Both benchmark runbooks read two-way by default.

How: our vLLM build (per-request attention) serves both readings, chosen per request. The shim marks each two-way read (vllm_xargs attention_mode: bidirectional); the engine prefills that prompt whole with the full-attention layers reading both ways and the sliding layers one-way, and keeps it out of the prefix cache. An unmarked read is stock one-way. Before its first answer the shim checks that the engine reads both ways: see Two-way start-up check.
Routing: a request reads two-way unless its record is longer than the two-way window, its summed two-way prompt is over the latency cap, or it has a Box question; those read one-way, whole. A request can choose with "kwargs": {"attention_mode": "two-way"} or "one-way".
Cost: a two-way read computes its whole prompt (no prefix cache), so a request with many questions about one long record costs more two-way than one-way; the latency cap sends those one-way.
Measured: with the Decision Index settings, two-way against the one-way fallback: text index +0.19 [-0.09, +0.46], vision +0.81 [-0.07, +1.67] (the same files and settings, only the reading differs).

Two-way reading on other GPUs

The Decision Index and JevBench runbooks do no GPU detection: they assume one RTX PRO 6000 and refuse a GPU under the profile's memory minimum. Only plain serve.sh (two-way by default: your own serving on our build, without a profile) looks its sizes up in the table below (serve_config.json engine_sizes). It reads the GPU's memory at start and picks the engine's sizes itself; only the listed GPU classes have sizes, and on any other GPU it stops and asks for them (or for READING=one-way). Set values always win.

GPU class (memory reported by `nvidia-smi`)	`MAX_LEN` (context)	`TWO_WAY_WINDOW`	`GPU_UTIL`
96 GB (92,000 to 100,000 MiB; one RTX PRO 6000 96 GB)	131,072	16,384	0.9
80 GB (76,000 to 86,000 MiB; one H100 80GB)	131,072	16,384	0.9

These sizes were not run with this release (the default aa setup's JevBench read above ran on the 96 GB row), and they are not the Decision Index setup: the Decision Index configuration above is one RTX PRO 6000 96 GB at its profile's own sizes (window 8,192).

The engine's prefill step (--max-num-batched-tokens) is the two-way window plus one token for each running sequence (MAX_NUM_SEQS, default 1024, vLLM's own default on these GPUs), so a window-sized two-way prompt runs beside a full batch of running answers instead of waiting for them to finish; the shim still reads nothing longer than the window two-way.

On another GPU, set MAX_LEN, TWO_WAY_WINDOW and GPU_UTIL (the same three sizes as for the runbooks). PLAN_SIZES=1 bash serve.sh prints the sizes it would use and starts nothing.

Long records

The two benchmark runbooks differ in three places: here, the conditional text re-read and the 1120 image re-read (both on in the Decision Index runbook only). SHIM_MAX_STATE_TOKENS=<n> sets the budget (it never exceeds MAX_LEN - 4,096); SHIM_SELECTOR=none sends every record whole and answers 422 when one is over the budget; SHIM_SELECTOR=basic is a simpler paragraph selector.
Read whole with refusal (SHIM_SELECTOR=none) is used by the Decision Index runbook and the aa and decision_index modes (so also by the fallback command), because the kit does not allow shortening inputs and its longest record fits the context. Every other setup selects an over-context record down to fit with BM25 and never refuses it (a record with no part matching the question, or one the selector cannot cut into fitting parts, is cut to its head and tail around a ... line, as in training, and answered): the default command (run_default.sh, up to 126,976 tokens whole) and the jevbench mode of plain serve.sh (3,000-token selection).

bf16 or FP8 weights

Default: bf16, the checkpoint as shipped (about 24 GB of weights on the GPU); serve_config.json says "quantization": "none", and the profiles start vLLM without --quantization.
Switch: QUANT=fp8 bash serve.sh serves vLLM's online FP8 (W8A8 dynamic) on an FP8-capable GPU. Not measured on the Decision Index with this release; on a held-out development set of 5,118 items its accuracy matched bf16 (0.8595 FP8 against 0.8587 bf16).

Image re-read (on in the Decision Index runbook)

An image choice or score question whose top probability is under a threshold is read again at a larger image budget (Gemma-4's 1120 soft tokens instead of the 560 the shim sends), and the answer is the mean of the two reads (yes/no image questions are read once).

Default: off (SHIM_IMAGE_REREAD_BUDGET 0). The Decision Index runbook turns it on at 1120 under 0.8, with SHIM_IMAGE_ORDERS=1 (no option-order second read for images, so the 1120 re-read is the only extra image read); in the measured vision read it re-read 4,104 of the 12,210 rows. The default command and the API setup keep it off: it reads the image a second time, and JevBench's speed score reads the 95th percentile.
Switch: SHIM_IMAGE_REREAD_BUDGET=1120 SHIM_IMAGE_REREAD_BELOW=0.8; /health reports the setting and counts the re-reads (auto.image_rereads with the two-way reading; images.reread_count on a one-way server).

Box questions

A box question ({"type": "box", "instructions": "the coffee mug in photo"}) answers with a bounding box in fractions of the image, its confidence, and the probability that the target is present. On by default: in normal serving and in both benchmark runbooks (run_decision_index.sh, run_default.sh), wherever images are read (there is no on/off setting). The box comes from one guided (JSON-schema) generation, so the engine needs structured outputs (vLLM has them on by default); an engine without them refuses a request with a box question with HTTP 422, and requests without one are unaffected. A text-only deployment refuses a box question with HTTP 400. box.mode "generate" (one constrained generation of the box) is the mode to use with this release.

ImageJevBench Box track: how to run

Ask a Box question through the package, not through a chat prompt. On a stock vLLM chat endpoint the weights answer with a box but are not trained to state a confidence for it: in this model the box's confidence is computed by the shim, from the model's own yes/no readout of the box drawn on the image ("does the red box tightly contain the target?"). Run the server, one request at a time on one GPU:

hf download h2oai/h2o-lightning-12b --revision v1.0 --local-dir h2o-lightning-12b
bash h2o-lightning-12b/run_default.sh        # prints DEFAULT SERVER READY on http://127.0.0.1:8741

One request per item, the image in the record and one box question that names it:

curl -s http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"photo": "data:image/jpeg;base64,<the image, unchanged>"},
  "questions": {"where": {"type": "box", "instructions": "the leftmost dog in `photo`"}}}'

Read answers.where.box (left, top, right, bottom, fractions of the image from the top-left corner) and answers.where.confidence (the readout's estimate that the box overlaps the target with IoU >= 0.5). If answers.where.found is false, the box is the whole image and the confidence is 0. present is whether the named object is visible; it is not the box's score.

The box confidence was checked on these weights on an out-of-distribution box set (RefCOCOg validation, 600 questions, board images excluded), served by the decision-index profile: calibration error 0.050 with the readout's own probability, accuracy at IoU 0.5 84.5 %, mean IoU 0.792. A fitted map (one or two parameters) lowered the error held out on one half and raised it on the other, so none is applied. We have not run this model on the Box track.

Other settings

The yes/no commit floor is off as shipped; SHIM_NOUL_FLOOR=<value in (0.5, 1)> turns one on (a yes/no answer then reported at no less than that on its chosen side; the answer never changes).
The calibration temperatures, fitted on these weights (see Calibration): one-way (serve_config.json) choice 1.12, score 0.7, yes/no 1.08, images 0.92. Two-way (serve_config_two_way.json) choice 1.12, score 0.7, yes/no 0.93, images 0.91; the decision-index profiles, and the aa and decision_index modes, read images at 0.91. SHIM_TEMPERATURE (every type), SHIM_TEMPERATURE_CHOICE / _SCORE / _NOUL and SHIM_IMAGE_TEMPERATURE override them.
serve.sh keeps vLLM's compile cache in one folder per reading mode, image setting and weight format (~/.cache/vllm/h2o-lightning-12b/<mode>-<images>-<weights>): vLLM 0.30 would otherwise reuse a graph compiled for another image setting (for example MAX_IMAGES=0) and fail at start-up. VLLM_CACHE_ROOT overrides it.
One-way (READING=one-way, no profile), serve.sh also starts vLLM with --mm-processor-cache-gb 0 (vLLM's front-end image-processor cache off; answers do not depend on it) and --allowed-media-domains media.invalid (vLLM never fetches a URL; the shim sends images as data). The two-way server, with or without a profile, keeps exactly the engine flags of the measured benchmark runs.
MAX_LEN, GPU_UTIL, VLLM_PORT, PORT and VLLM_ARGS adapt serve.sh to other hardware.

The API

Send requests to POST /v1/systemone. One request can carry several questions about the same state:

{"state": "Ticket 4471: since 09:10 the checkout page returns HTTP 500 for every customer; no orders have gone through in 40 minutes. Severity guide: P1 = a revenue path fully down; P2 = degraded but working; P3 = cosmetic.",
 "questions": {
   "priority":  {"type": "choice", "instructions": "Which priority does the severity guide assign?",
                 "criteria": {"p1": "Revenue path fully down", "p2": "Degraded but working", "p3": "Cosmetic"}},
   "all_users": {"type": "noul", "instructions": "The problem affects every customer.",
                 "criteria": {"true": "Affects everyone", "false": "Affects only some"}},
   "urgency":   {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["Low", "Medium", "High"]}}}

Each answer carries its probabilities and a confidence. usage.input_tokens counts the prompt head that the questions share once; usage.output_tokens is one label token read per answered question.

Images. Put each image in the request as a data:image/...;base64,... URI, anywhere inside state or in a top-level images list. Up to 4 images per request; image answers use their own temperature (serve_config.json, "image"). The shim also serves POST /v1/decisions (OpenAI's Decisions API shape, client.decisions.create in the openai Python package).

Status codes. An input the shim cannot take (more than 255 options, an unknown question type, a malformed question, more than 4 images, an image vLLM cannot load) gets HTTP 422; an empty request gets 400. A misconfigured vLLM (wrong served name, too small a context, a different chat rendering or tokenizer) is a 503, so a broken server stops a run instead of scoring it. An optional extra read (the text re-read, the image re-read, the reverse-order second read) that cannot be made never fails a request: the first answer stands, and effort_used and /health say so. On a server whose two-way start-up check did not pass, an explicit two-way request gets HTTP 400.

Tests (no GPU): python3 h2o-lightning-12b/test_shim.py and python3 h2o-lightning-12b/test_security.py (test_shim.py needs the tokenizers package for its serve.sh tests).

Limitations

Below the base model on some benchmarks. On PhishNChips (phishing decisions) this release scores under gemma-4-12B-it in every reading: verdict accuracy 80.3 % with the two-way reading and 81.9 % with the one-way fallback, against 83.5 % (differences -3.2 points [-4.8, -1.7] and -1.6 [-3.0, -0.1]; 2,000 paired draws each). It is the one Decision Index text benchmark where the shortfall is clear of noise; the others under the base model (listed in This release) are within it.
Class balance on two benchmarks. On RAGTruth (response-level hallucination) the model calls 43.4 % of the responses hallucinated where 34.9 % are (the base model: 38.8 %), although its F1 on the hallucinated class is higher than the base model's (0.800 against 0.765, +0.035 [+0.019, +0.052]). On Amazon ESCI it picks "Exact" for 55.3 % of the queries where 61.6 % are (the base model: 58.2 %), although its accuracy is higher (70.7 % against 66.3 %).
Evaluated mostly in English. Up to 255 options per question; a request with more gets 422.
The probabilities are calibrated for decision questions of this kind, not for open-ended text.
Not a chat model: it answers one decision per question and does not generate explanations.
No video input.
The engine's compile and kernel-tuning cache (VLLM_CACHE_ROOT) can differ between installs and shift probabilities slightly, occasionally flipping a near-tie answer; reusing the same cache reproduces results exactly.
The "I don't know" cutoff (aa mode) was fitted on our question mix; on much harder questions a higher cutoff scores better (see Modes).

Credits

Cygnet (blockbrain-ai/cygnet-recipe, commit 74f9555, MIT): the prompt layout and the one-token option-letter readout this shim implements. Its MIT copyright and permission notice are in LICENSE-cygnet. Cygnet credits the one-token readout approach to NInfer.
Header layout (the conditional re-read of the decision-index and decision-index-max profiles, and SHIM_LAYOUT=header): built token for token as FlexDM v1.0 builds it (MIT), a layout FlexDM takes from Will Cole (MIT); their MIT notices are in LICENSE-flexdm and LICENSE-wcole3. Its chat layout parts and the _index position annotation of long arrays are decider's (Apache-2.0). Reading the user turn twice follows shisa-ai/shisa-de-2.
Weights: built on Google DeepMind's gemma-4-12B-it (revision 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7, Apache-2.0).
Fine-tuning data: Improved using Qwen: part of the fine-tuning data (nvidia/OpenScienceReasoning-2) includes questions generated with Qwen2.5-72B-Instruct (Qwen License Agreement).
Serving: vLLM 0.30.0 (Apache-2.0), unmodified for the one-way fallback; the two-way reading runs on our vLLM build with per-request attention (h2oai/vllm, a modified vLLM 0.30.0, Apache-2.0). This repository ships no vLLM code; the Apache License 2.0 is in LICENSE-APACHE-2.0.

See NOTICE.

License

Non-commercial. See LICENSE.

Based on google/gemma-4-12B-it (Apache-2.0, Copyright Google); Apache-2.0 text: https://www.apache.org/licenses/LICENSE-2.0

The portions of the shim taken from Cygnet are under the MIT License in LICENSE-cygnet. The header layout's portions taken from FlexDM and Will Cole are under the MIT License in LICENSE-flexdm and LICENSE-wcole3; decider's under the Apache License 2.0 in LICENSE-APACHE-2.0.