Instructions to use h2oai/h2o-lightning-31b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2o-lightning-31b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="h2oai/h2o-lightning-31b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("h2oai/h2o-lightning-31b")
model = AutoModelForMultimodalLM.from_pretrained("h2oai/h2o-lightning-31b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2o-lightning-31b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2o-lightning-31b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-31b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/h2oai/h2o-lightning-31b

SGLang

How to use h2oai/h2o-lightning-31b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2o-lightning-31b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-31b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2o-lightning-31b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2o-lightning-31b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use h2oai/h2o-lightning-31b with Docker Model Runner:
```
docker model run hf.co/h2oai/h2o-lightning-31b
```

Accept the H2O Model Non-Commercial Licence to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is released under the H2O Model Non-Commercial Licence v1.1 (see the LICENSE file): non-commercial, non-production use only.

H2O-Lightning-31B

H2O-Lightning-31B is a 31B-parameter decision model from H2O.ai, built on google/gemma-4-31B-it. It answers typed decision questions about a record (a document, a ticket, a policy, a conversation), and about images that come with the record. It returns a probability for every option:

choice: picks one of a set of named options.
yes/no (noul): gives the probability that a statement is true.
score: gives an ordinal level on a stated scale, with its probability distribution.

This card describes v1.3.4 (main; pin --revision v1.3.4).

Leading on Benchmarks

Board results as posted for v1.2.4.

Vector SVG

#1 of 170 — Capability (open-weight Jev-class models; #5 of 201 Jev-class models, open or API), JevBench board of 2026-10-28, its row for release v1.0.7, measured 2026-10-22; ahead of Nace AI Drex Max (84.23), Liquid AI d1-ultra (80.48), OpenAI Decisions (73.50) and Microsoft-Decision-1 (70.78) (details)
#1 of 170 — Intelligence (open-weight Jev-class models; #3 of 201 Jev-class models, open or API), JevBench board of 2026-10-28, its row for release v1.0.7, measured 2026-10-22; ahead of Nace AI Drex Max (78.89), Liquid AI d1-ultra (69.70), Microsoft-Decision-1 (57.17) and OpenAI Decisions (56.90) (details)
#3 of 170 — Composite (open-weight Jev-class models; #15 of 201 Jev-class models, open or API), JevBench board of 2026-10-28, its row for release v1.0.7, measured 2026-10-22; ahead of Liquid AI d1 (72.82), Microsoft-Decision-1 (69.08), OpenAI Decisions (62.37) and Perplexity pplx-decider-flash (54.01) (details)
#1 of 46 — Capability (open-weight Jev-class models; #1 of 55 Jev-class models, open or API), ImageJevBench edition v0.3.0, page read 2026-10-29, its row for release v1.0.7; ahead of Nace AI Drex Max (72.47), GPT-6 Luna via OpenAI Decisions (62.87), Cloudflare Clef (58.08) and Liquid d1-3B (47.78) (details)
#1 of 46 — Intelligence (open-weight Jev-class models; #1 of 55 Jev-class models, open or API), ImageJevBench edition v0.3.0, page read 2026-10-29, its row for release v1.0.7; ahead of Nace AI Drex Max (66.96), Cloudflare Clef (47.40), GPT-6 Luna via OpenAI Decisions (45.00) and Liquid d1-3B (16.31) (details)
#3 of 46 — Composite (open-weight Jev-class models; #5 of 55 Jev-class models, open or API), ImageJevBench edition v0.3.0, page read 2026-10-29, its row for release v1.0.7; ahead of Nace AI Drex Max (44.65), Cloudflare Clef-Flash (20.41), GPT-6 Luna via OpenAI Decisions (17.30) and Liquid d1-3B (4.10) (details)
#3 — Text (board score, its private sets, the board's own tie groups; tied for 3rd of 244 entries), Decision Index 0.4.1, board of 2026-10-28, its row for release v1.1; ahead of Nace.AI Drex Max v1 (66.60), Cloudflare clef (54.92) and Perplexity Decider Flash (54.29) (details)
#13 of 61 — Vision (full score, public and private sets, among every entrant), Decision Index vision 0.4, board of 2026-10-28, its row for release v1.0.7; ahead of Perplexity Decider Flash (66.02) and Cloudflare clef-omni (59.54) (details)

This release

Adaptive reasoning for requests in the aa mode (the default of serve.sh): the one-pass read also returns the model's own activations, a small head decides whether the request would gain from deliberation, and if so the same weights on the same engine think about it before answering (see Adaptive reasoning). The Decision Index and JevBench runbooks never use it.

This card describes v1.3.4 (main; pin --revision v1.3.4). Compared with v1.3.0 (the table compares v1.3.0's weights with the weights this release ships, both read by the decision-index profile), on the Decision Index 0.3 public suites, both read with the decision-index profile on one NVIDIA RTX PRO 6000 (96 GB) per server (the v1.3.0 column is v1.3.0's own read, made on its release candidate build: the same weights and the same Decision Index settings as the published v1.3.0):

Decision Index 0.3 public	v1.3.0	this release	difference
text index	70.05	70.25	+0.20 [+0.06, +0.35]
vision index	76.60	76.75	+0.16 [-0.09, +0.42]

The Decision Index scores on this card describe the Decision Index profile files of this repository's commit e97edc4ef9bb3e85e5f2aea17e2008df13f251bf (run_decision_index.sh and the shim's decision-index profile), which this commit keeps byte for byte apart from the profile's engine pin: the engine those files install is our fork at commit a36449a18b4465b4f70fa078f5403032de414de8 (the build that also returns the activations for adaptive reasoning), whose decision-index reads agree with the previous build's: 6,697 of 6,762 answers agree (99.0%), max probability difference p99 0.047 (2,000 Decision Index requests).

The text difference is a bootstrap over the kit's case groups (10,000 draws); the vision difference is paired over the 12,210 public vision rows, resampled within each benchmark (10,000 resamples). The probability constants are v1.3.0's (see Calibration).

How to run it (recommended setup first):

Recommended: our vLLM build, two-way reading. Our h2oai/vllm fork with two-way support (installed over the matching stock vLLM wheel), with the shim (lightning_31b_shim.py) in front, reads each request two-way (one-way for a long or heavy request). Three profiles:
- decision-index, the Decision Index runbook (bash h2o-lightning-31b/run_decision_index.sh): every record read whole, a text choice or yes/no answer under 0.90 read a second time (the two reads averaged), an unsure image question read again at 1120 soft tokens.
- two-way, for JevBench and for API serving with one read per question (bash h2o-lightning-31b/run_default.sh, or serve.sh --profile two-way).
- decision-index-max, opt-in: the same re-read below 0.95.
Default configuration (AA mode): bash h2o-lightning-31b/serve.sh with no settings, on our build: the Decision Index setup read two-way, plus the "I don't know" rule (see Default configuration (AA mode)).
Fallback: upstream (stock) vLLM 0.30.0, not our fork, one-way reading, for a server without our fork: READING=one-way bash h2o-lightning-31b/serve.sh (see the fallback). Not measured on this release.
Modes: each request is answered in one of three modes (jevbench, decision_index, aa), named in the request or set by the server's default; see Modes.

JevBench (in-house, the 231 public items)

Measure (this release, `bash h2o-lightning-31b/run_default.sh`, one RTX PRO 6000)	Value
JevBench, 231 public items	216 / 231 (0 errors)
latency, the 231 request bodies sent once each, one at a time (client wall time; warm-up requests not counted)	median 41.1 ms, mean 107.6 ms, 80th percentile 116.1 ms
input tokens per decision	649.7 (mean `usage.input_tokens`)

Our own client sent each item once, one at a time; the board computes Intelligence, Calibration and Cost on its own items, so no board score is quoted here. This latency is for these 231 JevBench requests in jevbench mode (run_default.sh's default); it is not comparable with latency figures on earlier cards, which used other request samples.

Calibration (constants fitted on v1.3.0's weights)

Every temperature and the yes/no floor were fitted on v1.3.0's weights, and this release keeps them, from Decision Index 0.3 public per-row reads (halves split by a hash of each case, each value accepted only when both halves agree), and confirmed on a fresh set where one exists:

constant	value	basis
choice temperature, one-way reads	1.28	fitted (halves 1.29 / 1.27); in-sample
choice temperature, two-way reads	1.13	fitted (halves 1.12 / 1.13); confirmed on SuperGPQA (1,990 clean questions): calibration error 0.219 -> 0.100
yes/no temperature, two-way reads	1.07	fitted (halves 1.06 / 1.08); in-sample
yes/no temperature, one-way reads	0.6	not fitted (too few one-way yes/no rows)
score temperature	0.7	not fitted (no Decision Index score rows)
image temperature, decision-index profiles	0.96	fitted (halves 0.97 / 0.94); confirmed on MMStar (1,477 questions): calibration error 0.178 -> 0.074
image temperature, other profiles (`serve_config.json` image block)	0.9	not refitted
yes/no floor (both readings)	0.801	refit not accepted (halves disagree), kept
"I don't know" cutoff (`aa` mode)	0.80	fitted on a real-"I don't know" read of v1.3.0's weights; see Modes

The confirmation sets were decontaminated against v1.3.0's training data first (SuperGPQA: 1,990 clean questions of 2,000; MMStar: 1,477 items after removing 16 that matched training question text).

The image re-read of the decision-index profiles is gated on the tempered top probability (under 0.8), so the image temperature also sets how often it fires: at 0.96 it re-read 2,956 of the 12,210 public vision rows (24.2 %).

Modes

Each request is answered in one of three modes. A request names its mode in "mode" (or kwargs.mode); a request without one gets the server's default: decision_index under run_decision_index.sh (the decision-index profile), jevbench under run_default.sh (the two-way profile), and aa for serve.sh with no profile.

mode	`"mode"`	what it does
JevBench	`"jevbench"`	the server's own settings (under `run_default.sh`: one read per question, a record over 126,976 tokens cut down by BM25)
Decision Index	`"decision_index"`	the `decision-index` profile's reading: every record read whole up to the context, the conditional text re-read under 0.90, the image re-read at 1120 under 0.8
AA (`"auto"` is the same)	`"aa"` or `"auto"`	Decision Index, plus the "I don't know" rule: a choice question with an explicit "I don't know" option answers it when its top real answer, renormalised without that option, is under 0.80; plus adaptive reasoning: a single-question choice request without images gets a thinking pass on the same engine when the model's own activations say it would gain (see Adaptive reasoning)

The "I don't know" cutoff 0.80 was fitted on v1.3.0's weights from a real-"I don't know" read (4,398 scored questions; the lowest cutoff within noise of the best, 0.85, on our questions); the AA index gain over always answering is about +8.5. On easier families (MMLU, MMLU-Pro) 0.80 gains on 0.90 (+0.33 index); on harder ones (GPQA, HLE, about 37 % accuracy) it loses (-0.32). If your questions are much harder than ours, a higher cutoff (0.90) would score better. Every response says its mode in effort_used.mode; /health lists the modes and their settings.

Default configuration (AA mode)

bash h2o-lightning-31b/serve.sh with no settings starts the AA setup: the Decision Index setup read two-way on our vLLM build, plus the "I don't know" rule and adaptive reasoning. A request that names no mode is answered in aa mode.

What it reads. The decision_index mode's settings: every record read whole (up to 126,976 tokens, or the context minus 4,096 on a GPU with a smaller context; a longer record is refused with HTTP 422, never cut), a text choice or yes/no answer under 0.90 read a second time and the two reads averaged, an unsure image question read again at 1120 soft tokens. Each request is read two-way, and one-way, read whole, when its record is longer than the two-way window or its two-way prompts add up to more than 4,096 tokens, or it has a Box question (see Two-way reading).
The "I don't know" rule. It applies only to a choice question that offers an "I don't know"-type option. The answer is read as in decision_index mode; then the real options (every option except the "I don't know" one) are renormalised to add up to 1. If the most likely real option is under 0.80, the answer is the "I don't know" option (reported at 0.80); otherwise the answer is unchanged. Every other question is answered exactly as in decision_index mode. Each answer to such a question reports the rule in idk (the option, the cutoff, the top real probability and abstained).
Which options count. An option whose text (or name), lower-cased, with a typographic apostrophe read as ' and end punctuation dropped, is one of serve_config.json idk.labels: "i don't know", "i do not know", "don't know", "do not know", "dont know", "idk", "not sure", "i'm not sure", "i am not sure", "unsure", "no idea". A question with two such options gets no rule.
Adaptive reasoning. With the one-pass read the engine also returns the model's residual activations at the last prompt token (two layers); the shipped head (router_head.npz) decides from them, in about a millisecond on the CPU, whether this request would gain from deliberation. When it would, the shim asks the same weights on the same engine a second time with thinking enabled (a budget of 2,560 thinking tokens, the same on any load or hardware; at most 4 such passes per engine at once, a further request waiting its turn: an escalated request is never shortened, skipped or replaced because of load), reads the answer from the analysis and returns that decision; the analysis text is never returned. Only single-question choice requests without images are considered; multi-question requests, yes/no questions and image requests keep the single pass. What it costs and gains: Adaptive reasoning.
Start it. One command, on one GPU of a measured class (the per-GPU table: 96 GB and 80 GB), in the Python environment that holds our vLLM build (run_decision_index.sh installs it into ./di-venv on its first run):
```
bash h2o-lightning-31b/serve.sh
```
Without our build, or on a GPU without measured sizes, it still starts, but reads every request one-way (as READING=one-way does) and says so loudly: a warning at start naming the reason and both fixes (install the build, or set MAX_LEN and TWO_WAY_WINDOW for that GPU; or start READING=one-way bash h2o-lightning-31b/serve.sh to choose one-way and silence the warning), and /health shows "reading": "one-way (fallback: <reason>)" with the same words under "warnings". READING=two-way asks for two-way explicitly and stops instead when it cannot be served. If the engine is our build but its two-way start-up check does not pass, the server reads every request one-way and says so (see Two-way start-up check).
Turn it off. Per request: "mode": "decision_index" (the same reads without the rule) or "mode": "jevbench". Per server: bash h2o-lightning-31b/serve.sh --profile decision-index (default mode decision_index; the decision-index profile, sized for one RTX PRO 6000) or --profile jevbench (default mode jevbench). Adaptive reasoning alone: ADAPTIVE_REASONING=0 bash h2o-lightning-31b/serve.sh keeps the aa mode (the reads and the rule) without the thinking pass; per request, "reason": {"enabled": false} does the same for that request.
Check it. curl -s http://127.0.0.1:8741/health shows "default_mode": "aa", "reading": "two-way (verified)" ("one-way (fallback: <reason>)" when it serves one-way instead, as above), and under "modes", "aa", the cutoff "idk_cutoff": 0.8. Every response says its mode in effort_used.mode and its reading in effort_used.attention_mode. The ready line of serve.sh ends with "adaptive reasoning on (layers 30,40, threshold 0.0423, budget 2560, modes aa)" (or "adaptive reasoning off" with the reason); a response that went through the thinking pass carries effort_used.router (escalated: true, the head's gain) and effort_used.reasoned (status ok, finish stop or length = the token budget reached, the thinking token count, queued_s); every other aa response carries effort_used.router with escalated: false or the reason it was not considered.
Measured. bash h2o-lightning-31b/serve.sh with nothing set, on one RTX PRO 6000 from this release's files (the two-way start-up check verified, adaptive reasoning on): JevBench's 231 public items, each sent once, one at a time, in the server's default aa mode: 224 / 231 (0 errors); latency median 47.9 ms, mean 4468.0 ms, 80th percentile 335.3 ms (client wall time; warm-up requests not counted; an escalated request takes the thinking pass, see Adaptive reasoning); 649.7 input tokens per decision (mean usage.input_tokens). Without adaptive reasoning (ADAPTIVE_REASONING=0 bash h2o-lightning-31b/serve.sh, the same files and weights): 218 / 231 (0 errors); latency median 43.8 ms, mean 126.8 ms, 80th percentile 259.6 ms; 649.7 input tokens per decision.

Two-way start-up check

Before its first answer a server on our vLLM build checks that the engine really reads two-way: three short pairs of requests that share a prefix and differ after it are read flagged (two-way) and unflagged (one-way), and the unflagged reads once more. Two-way is served only when the flagged prefix moves with the later text (at least 0.1 in log-probability) while the unflagged prefix does not (at most 0.001) and the repeated unflagged read agrees with itself. If the check does not pass, the server does not refuse: it serves every request one-way, says so in /health (two_way_check.verified false, with the reason, and reading "one-way (fallback: <reason>)"), in a warning in its log every 10 minutes and in every answer (effort_used.attention_reason "two-way unavailable (start-up check failed)"), and answers an explicit two-way request with HTTP 400. The only refusal (HTTP 503) is a deterministic engine whose one-way reading sees later tokens. The benchmark runbooks stop instead of serving the fallback (their /health check expects two_way_check.verified). A fixed set of reference probes is also read at start and reported in /health; it does not gate: FP8 compiles of the same weights move it by a few hundredths from start to start (one start of v1.3.0 read 0.1348 on probe y05 on a healthy server).

JevBench board results

Board results as posted for v1.2.4.

H2O-Lightning-31B and our H2O-Lightning-4B are neck and neck at the top of the JevBench open-weights composite (73.44 and 73.82).

On the public JevBench board (board v1.7.47, read 2026-10-22; the API's release v1.6.40), the board's own run of this model scores:

Intelligence	Calibration	Speed	Cost	Capability	Composite
80.15	90.19	89.84	50.57	85.17	73.44

Every placement below ranks the board's Jev-class models only: those within its cost and latency limits (166 models, 139 of them open-weight, not counting the board's two reference rows).

Intelligence: #1 of the board's 166 Jev-class models, open or API (next: Chon-1.0-12B, 79.86).
Capability (the board's mean of Intelligence and Calibration): #2 of 139 open-weight Jev-class models (behind Chon-1.0-12B, 85.34) and #3 of 166 Jev-class models (an API model, kettei-fat, 86.31, is first).
Composite (the board's official score: the harmonic mean of the four axes, weighted equally): #2 of 139 open-weight Jev-class models, behind our H2O-Lightning-4B (73.82), and tenth of 166 Jev-class models, API included.

The board estimates Cost from the base model's reference price per token and the tokens our server reports.

Screenshots of the board, 2026-10-22 (open-weight models, then all models with the board's API rows shown):

ImageJevBench

Board results as posted for v1.2.4.

On the board's ImageJevBench page (edition v0.3.0, read 2026-10-22), the board's own run of this model with images scores:

Intelligence	Calibration	Speed	Cost	Capability	Composite
75.61	86.15	83.80	43.70	80.88	51.22

Every placement below ranks the edition's Jev-class models only (54 models, 45 of them open-weight).

Capability: #1 of 54 Jev-class models (next: kettei-fat, 75.89).
Intelligence: #1 of 54 Jev-class models, open or API (next: Sage 1.4 + images, beta, 68.13).
Composite: #2 of 45 open-weight Jev-class models (behind Imajev-4B, 63.71) and fourth of 54 Jev-class models, held back by the board's Cost score for this self-hosted run (43.70, from measured GPU time at $1.19 per GPU-hour: $0.075 per 1,000 decisions). A Cost score under 50 scales the composite by (Cost / 50)², here 0.76 (without it, 67.06); the other three axes are all above 75.

Screenshots of the ImageJevBench page, 2026-10-22:

Decision Index board

Board results as posted for v1.2.4.

Vector SVG · Every bar is the Decision Index board's own number for that entry, ours included, read from a saved copy of the board (Hugging Face Space multimodalart/jev-decision-index, as generated on 2026-10-22; text Decision Index 0.4.1, vision 0.4). Each panel shows every entry the board ranks above H2O-Lightning-31B and 4 below it, in the board's order. Text: the public index and the board's text score, which ranks the board: its private sets only (same skills 60%, new domains 40%, each equated to the public scale; public weight 0%). Vision: the public and the private score; the board ranks by their mean. On text our 31B is 16th (66.58; the board groups entries into 0.9-point tie bands and lists ours as tied), and its new-domains score, 55.20 before equating, is the lowest of the entries shown. On vision it is 11th (70.56): its public score, 76.51, is fourth on the board, and its private score, 64.60, is the lowest of the entries ranked above it (11.91 points lower than its public score).

Recommended: how to run for the Decision Index

For Decision Index maintainers: run it this way. The engine sizes are for one NVIDIA RTX PRO 6000 (96 GB), the GPU every number on this card was measured on. One shipped command installs the pinned engine, serves the model with the settings the expected numbers below were measured with, checks the running server against every value of the profile, and only then says it is ready. There is nothing to choose and no setting to type: run the one command in step 2.

What it serves: our vLLM build, reading each request two-way (the model's trained two-way reading, full-attention layers looking both ways) and switches a request to one-way, read whole, when its record is longer than the two-way window or its two-way prefill would be slow. No record is ever cut, selected or summarised.

Machine. One NVIDIA RTX PRO 6000 (96 GB), NVIDIA driver 580 or newer (the engine install brings PyTorch's CUDA 13.0 build), Linux, Python 3.10 to 3.12 with its venv module, a C compiler, curl and git. Disk: about 63 GB for the download and about 12 GB for the engine's environment (and about 17 GB to rebuild the vision suite). On a minimal image such as pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime:

apt-get update && apt-get install -y gcc g++ curl git cabextract && pip install uv

1. Download this release. The repository is gated: accept the licence on the model page, then log in.

pip install -U huggingface_hub && hf auth login
hf download h2oai/h2o-lightning-31b --revision v1.3.4 --local-dir h2o-lightning-31b

2. Start the server with the one shipped command:

bash h2o-lightning-31b/run_decision_index.sh

What it installs. On the first run only, our vLLM fork with two-way support into ./di-venv (beside where you run it): the matching stock vLLM 0.30.0 wheel (with the measured PyTorch 2.13.0, transformers 5.20.0 and torchcodec 0.17.0) and the changed files of h2oai/vllm at commit a36449a18b4465b4f70fa078f5403032de414de8 (the per-request two-way attention build with the residual-activation return of Adaptive reasoning), each checked against engine/vllm_per_request_attention.SHA256SUMS. Every later run checks the version and every file again, and refuses a changed engine. No package is installed outside ./di-venv; vLLM's compile caches go under ~/.cache/vllm/h2o-lightning-31b/, one folder per engine build, reading mode, sizes, image setting and weight format, so no setting reuses another's compiled graphs (VLLM_CACHE_ROOT chooses another folder; both runbooks pass it through).

It does four things, and stops with REFUSE: and the reason on any mismatch:

a. checks the NVIDIA driver (580 or newer), Python, a C compiler, curl and git, and that the shipped files are this release's (engine/package.SHA256SUMS); b. installs or checks the engine, as above; c. runs serve.sh --profile decision-index in that environment: the GPU's memory checked against the profile's GPU, vLLM started with the arguments the shim prints, the shim started with --profile decision-index. The profile's settings are below. Every setting in this table is a default that --profile decision-index (and run_decision_index.sh, which uses it) applies by itself: you set none of them when you start with the Decision Index option, and the shim refuses any of them from the environment.

setting	value
engine	our vLLM build, reading each request two-way or one-way
weights	FP8 (vLLM's online FP8 of the bf16 checkpoint, `--quantization fp8`)
context (`MAX_LEN`)	131,072 tokens, every Decision Index record read whole
two-way	a request whose questions' two-way prompts add up to at most the latency cap below, and whose record fits the two-way window (8,192 tokens; `TWO_WAY_WINDOW`)
one-way, read whole	any other request: one whose summed two-way prompt is over 4,096 tokens (the latency cap)
readout orders	one read per question
conditional re-read	a text choice or yes/no answer whose top probability is under 0.90 is read a second time, with the request written out twice, and the two reads are averaged; only for requests of up to 2,000 tokens
images	one read at Gemma-4's 560-token image budget; a choice or score image question whose top probability is under 0.8 is read again at 1120 and the two reads averaged
Box questions	on (as in normal serving; no setting): one guided generation per box question, read one-way
record (`state`)	read whole up to 126,976 tokens; a record that does not fit is refused with HTTP 422, never cut
temperatures and yes/no floor	one-way choice 1.28, score 0.7, yes/no 0.6; two-way 1.13, 0.7, 1.07; images 0.96 in both readings; floor 0.801
default mode	`decision_index` (a request may ask for `aa` or `jevbench`)

For reference only (the profile sets them; do not export them): the re-read row is SHIM_REREAD_BELOW=0.9, SHIM_REREAD_COMBINE=avg, SHIM_REREAD_MAX_TOKENS=2000, SHIM_REREAD_LAYOUT=header; the image row is SHIM_IMAGE_REREAD_BUDGET=1120, SHIM_IMAGE_REREAD_BELOW=0.8, SHIM_IMAGE_ORDERS=1. See Conditional re-read and Image re-read.

d. waits for the server and compares its /health with the profile, field by field (lightning_31b_shim.py --profile decision-index --check-health): "ok": true; profile.name "decision-index"; serving.attention_mode "auto"; the two-way start-up check verified (two_way_check.verified); auto.engine "one", auto.max_total_tokens the cap and auto.two_way_window the window; max_state_tokens 126976 and selector "none"; the temperatures of both readings (one-way choice 1.28, score 0.7, yes/no 0.6; two-way 1.13, 0.7, 1.07; images 0.96 in both); both yes/no floors 0.801; readout orders "off" (one read); the image re-read {"budget": 1120, "below": 0.8} in both readings, with image orders 1.

Before it reports ready, serve.sh sends one warm-up request down every path this configuration serves (warmup.py: two-way, several questions at once, a request over the latency cap, a record over the two-way window, a long record, an image at the 560 budget and at the 1120 re-read budget), so no real request pays a first-use compile; until then /health answers 503 with "warming": true. When it prints DECISION INDEX SERVER READY on http://127.0.0.1:8741, the server is the measured configuration. Every answer states which reading served it (effort_used.attention_mode and effort_used.attention_reason).

3. Run the Decision Index kit (edition 0.3) against the server, from a second shell (the server keeps the first) and in its own environment:

uv venv --python 3.12 kit && VIRTUAL_ENV=kit uv pip install "decision-index[vision] @ git+https://github.com/apolinario/decision-index@f8552a1" pillow==13.0.0 numpy
kit/bin/decision-index suite download --edition 0.3 --dir suite-0.3
kit/bin/decision-index run --edition 0.3 --suite-dir suite-0.3 --engine http --option base_url=http://127.0.0.1:8741 --option model=h2oai/h2o-lightning-31b --out runs/text
kit/bin/decision-index score --edition 0.3 --suite-dir suite-0.3 --results runs/text/results.jsonl --engine http --out runs/text/score
kit/bin/decision-index vision run --dir <vision suite dir> --engine decide --option base_url=http://127.0.0.1:8741 --option path=/v1/systemone --option protocol=decisions --option model=h2oai/h2o-lightning-31b --option max_pixels=1638400 --workers 16 --out runs/vision
kit/bin/decision-index vision score --dir <vision suite dir> --results runs/vision/results.jsonl --out runs/vision/score

<vision suite dir> is WORK/vision, written by the rebuild in the last item below, or the pinned vision suite you already hold (decision-index vision verify must pass on it).
One text run in a single process takes days on one GPU. The measured text run was split, as a speed-up only: the kit's own row list (rows unchanged) into the 16 files of the shipped di_tools/di_text_sharded.sh, read first by one server of this package and then by two (one per GPU, each file resumed by the kit from its own results), the untouched results concatenated and scored once by decision-index score. di_tools/di_text_sharded.sh (16 files round-robin against one server) replaces the text run and score lines above (it writes runs/text/results.jsonl and scores it into runs/text/score). A different split batches requests differently on the engine, so individual answers can move as between two compiles. It needs decision-index on the PATH, e.g. the kit environment activated:
```
. kit/bin/activate
bash h2o-lightning-31b/di_tools/di_text_sharded.sh suite-0.3 http://127.0.0.1:8741 h2oai/h2o-lightning-31b runs/text 16
```
Build the vision suite with the kit's own builder, giving it Microsoft's core-fonts Arial Bold: the kit draws Mind2Web's box labels in Arial Bold, and only that font reproduces the pinned images. The file is arialbd.ttf from arial32.exe (the core-fonts package Ubuntu's ttf-mscorefonts-installer downloads, from https://downloads.sourceforge.net/corefonts/arial32.exe; sha256 of arial32.exe 85297a4d146e9c87ac6f74822734bdee5f4b2a722d7eaa584b7f2cbf76f478f6, of arialbd.ttf 4044aa6b5bebbc36980206b45b0aaaaa5681552a48bcadb41746d5d1d71fd7b4):
```
curl -L -o arial32.exe https://downloads.sourceforge.net/corefonts/arial32.exe   # check its sha256 against the one above
cabextract -L -F '*.ttf' arial32.exe        # gives arialbd.ttf; check its sha256 against the one above
kit/bin/decision-index suite rebuild --vision --work WORK --font "$PWD/arialbd.ttf"
kit/bin/decision-index vision verify --dir WORK/vision
```
vision verify must pass on all 11 benchmarks; the vision command above then runs as written. Without that font the kit falls back to Pillow's default font, Mind2Web's image bytes differ from the pin, and the vision run stops before reading anything. For the measured suite only Mind2Web was rebuilt with this font, by the kit's own builder (the other ten sets come from an earlier build of the same kit), and the suite passed vision verify on all 11; a from-scratch suite rebuild --vision --font of all eleven was not run end to end.

How to tell it is right.

The command printed DECISION INDEX SERVER READY on http://127.0.0.1:8741: it says that only after the warm-up and after every /health value above matched the profile.
The kit's scores land at the expected numbers: Decision Index 0.3 public text index 70.25 and vision 76.75, measured exactly this way (the decision-index profile that run_decision_index.sh starts, one RTX PRO 6000 per server) on this release's files. The engine reads in FP8 with many requests at once, and the engine's compile and kernel-tuning cache can differ between installs and shift probabilities slightly, occasionally flipping a near-tie answer: two fresh compiles of v1.3.0, read on the same 203 vision rows, differed in 7 answers. A rerun on a fresh install can differ in the second decimal of the index.
In the kit's results, every response carries effort_used.attention_mode (bidirectional for a two-way read, causal for one-way), and an image response re-read at 1120 carries effort_used.image_reread.

What not to change.

No environment variables for the server: the command starts it with a clean environment (only the GPU selection, CUDA_VISIBLE_DEVICES, the library path, PATH, HOME, HF_HOME and the three sizes for another GPU pass through), and the shim refuses a setting of its profile from the environment.
No edits to the downloaded files: the command checks the files it serves with (serve.sh, the shim, its configs, the warm-up, the engine's checksums) against engine/package.SHA256SUMS and refuses a changed one.
Not stock vLLM: the one-way fallback below is a different configuration with its own numbers, not measured on this release.
Not run_default.sh for the Decision Index: it is the two-way profile, which leaves out the 1120 image re-read and would cut a record over 126,976 tokens down with BM25 (no Decision Index record reaches that); this card's Decision Index numbers were not checked against it.
The kit commands as written in step 3, with max_pixels=1638400 for vision.

Latency. Not measured with this runbook on this release. The serial latency measured on this release is the JevBench one above (a different request set and mode). The board's latency rule (median, mean and 80th percentile each at most 1,000 ms on its own GPU) has not been checked for this runbook.

Troubleshooting.

Two copies on one machine: run_decision_index.sh always uses ports 8000 (vLLM) and 8741 (the shim), so run one copy per machine.
REFUSE: no C compiler or serve.sh needs a C compiler: vLLM compiles the model at start-up. Install gcc g++ (and curl git) as in Machine above.
REFUSE: ... is not the pinned per-request attention build: ./di-venv holds another vLLM. Remove the folder and run the command again; it reinstalls.
the two-way warm-up through the shim failed: the engine answered the two-way start-up check like a one-way engine, which is what stock vLLM does (it ignores the per-request flag). The cause is an engine that is not the pinned build; remove ./di-venv and run again.
Could not find nvcc at the first request: FlashInfer's sampler compiles itself on first use and needs the CUDA toolkit, which minimal images lack. The decisions never sample, so serve.sh turns that sampler off (VLLM_USE_FLASHINFER_SAMPLER=0); the error means it was turned back on in the environment.
CUDA out of memory at start: the engine takes 92% of the GPU's memory (GPU_UTIL 0.92); nvidia-smi shows what else holds memory on it.
the warm-up failed (above): a warm-up request (warmup.py, one per path) did not answer 200; the line above it names the path and the server's answer. A path that fails the warm-up would fail real requests the same way.
REFUSE: the server is not the decision-index profile's configuration: the lines under it name each /health field and the value the profile expects; a changed file in the download is the usual cause (download the release again).
This profile is for one NVIDIA RTX PRO 6000 (96 GB) and this GPU has ... MiB: the GPU has less memory than the profile's; see Another GPU below.
profile decision-index sets ... itself or ... would change the measured configuration: a SHIM_* variable is set in your environment; unset it (the profile sets the shim's settings itself).
the shipped configs are not the measured ones: serve_config.json or serve_config_two_way.json was changed; download the release again.
shim: ENGINE CHECK FAILED (and HTTP 503 on every request): vLLM was not started as the profile needs; the message names the command to start it with.

Other ways to start the same server (not needed for a measured run). run_decision_index.sh above is the measured run; the notes below are for maintainers who start the pieces themselves.

Where the settings live. Every Decision Index value is the shim's decision-index profile (lightning_31b_shim.py, PROFILES): the shim's settings, the vLLM engine's arguments and sizes, the vLLM version and the per-request attention commit. The script holds none of them; it installs the engine and starts vLLM and the shim with what the profile says. If you run the shim yourself, pass --profile decision-index: start vLLM (with our per-request attention build installed, as below) with the arguments the shim prints, then the shim with the profile. At start the shim checks the engine (the served model and its context, the chat rendering, the per-request attention build answering both readings) and refuses every request, naming the command to start vLLM with, until it matches; the weight format and memory settings are not visible to it, so start vLLM with the printed arguments as they are. bash h2o-lightning-31b/serve.sh --profile decision-index is the script's server without the install step (for an environment that has the engine).

. di-venv/bin/activate   # the engine's environment, as run_decision_index.sh installs it
python3 h2o-lightning-31b/lightning_31b_shim.py --profile decision-index --print-vllm-args \
  | VLLM_USE_FLASHINFER_SAMPLER=0 xargs -d '\n' vllm serve h2o-lightning-31b --host 127.0.0.1 --port 8000 &
python3 h2o-lightning-31b/lightning_31b_shim.py --profile decision-index --tokenizer h2o-lightning-31b/tokenizer.json

Another GPU. The profile's sizes are the RTX PRO 6000's; the command refuses a GPU with less memory and has no other GPU logic. On another GPU set the three sizes yourself, e.g. for one 80 GB H100: MAX_LEN=81920 TWO_WAY_WINDOW=16384 GPU_UTIL=0.92 bash h2o-lightning-31b/run_decision_index.sh. A record is then read whole only up to MAX_LEN - 4,096 = 77,824 tokens, and a longer one is refused with HTTP 422 and scored wrong; /health lists the sizes under profile.overrides. This release's numbers were measured at the profile's own sizes on one RTX PRO 6000.

Recommended: how to run for JevBench

Follow these steps literally; there is nothing to choose. The machine, the download and the engine are the Decision Index block's, and so are the settings, with two differences: a record over 126,976 tokens is cut down by BM25 selection instead of refused, and images get no 1120 re-read (see Two-order reading).

Machine and download as in steps "Machine" and 1 of the Decision Index block. This runbook is the shim's two-way profile; it is also the default server for general use on our vLLM build.

Start the server with the one shipped command:

bash h2o-lightning-31b/run_default.sh

It does the same steps as run_decision_index.sh with the two-way profile (the GPU check, the pinned engine, every setting fixed, /health compared field by field) with BM25 selection over 126,976 tokens and no conditional re-read, and prints DEFAULT SERVER READY on http://127.0.0.1:8741 when the server is the measured configuration. Every setting in this table is a default that the two-way profile (and run_default.sh, which uses it) applies by itself: you set none of them.

setting	value
engine	our vLLM build, reading each request two-way or one-way, FP8 weights
reading	two-way by default; one-way, read whole, for a record over the two-way window or a request over the latency cap
readout orders	one read (`off`)
temperatures and yes/no floor	one-way choice 1.28, score 0.7, yes/no 0.6, images 0.9; two-way 1.13, 0.7, 1.07, 0.9; floor 0.801; default mode `jevbench`
record (`state`)	read whole up to 126,976 tokens; a longer record is cut down to fit by BM25 selection (`SHIM_SELECTOR=v2`), never refused (a record with no part matching the question keeps its head and tail around a `...` line instead, `effort_used.state_selection` `"no match; head+tail"`)
images and Box questions	on (as in normal serving; no setting): images at Gemma-4's 560-token budget, read once; one guided generation per box question, read one-way

Run JevBench with the JevBench harness against http://127.0.0.1:8741/v1/systemone (model name h2oai/h2o-lightning-31b), one request at a time: the board's speed rule reads serial latency. The numbers below came from our own client (not the board's harness) sending each of the 231 public items once, serially, to that endpoint.

Expected numbers: see JevBench above (this command).

Fallback: stock vLLM, one-way only

Without our fork, upstream (stock) vLLM 0.30.0 serves the model one-way (no two-way reading), with the same shim and the same files, started with READING=one-way (without it, serve.sh serves the same one-way server but warns that it is not the default two-way reading). This release has no benchmark numbers in this configuration; use our vLLM build for the benchmark numbers above. This one-way configuration with the Decision Index settings below was run once on v1.3.0 (stock vLLM 0.30.0, one NVIDIA L40S 48 GB with MAX_LEN=13312 GPU_UTIL=0.95 for that GPU): it started, /health reported the settings listed in step 3, and 8 Decision Index text requests were answered one-way in jevbench mode. serve.sh's warm-up sends a record longer than 8,192 tokens, so the context (MAX_LEN) must be at least about 12,800 tokens for it to start.

Hardware. Any FP8-capable GPU with at least 80 GB (an H100 80GB, for example); not measured on this release.

1. Install upstream vLLM 0.30.0 (not our fork) in a fresh Python 3.12 virtual environment (the measured environment: vllm 0.30.0, torch 2.13.0, transformers 5.20.0, torchcodec 0.17.0, Python 3.12), and download this release. The machine also needs a C compiler and curl: vLLM compiles the model at start-up, and serve.sh stops with a message when either is missing. Minimal images such as pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime have neither (apt-get install -y gcc g++ curl). The repository is gated: accept the licence on the model page, then log in (hf auth login).

python3.12 -m venv .venv && . .venv/bin/activate
pip install "vllm==0.30.0" "torchcodec==0.17.0"
hf auth login
hf download h2oai/h2o-lightning-31b --revision v1.3.4 --local-dir h2o-lightning-31b

2. Serve with the shipped script and exactly these two settings, which read every record whole:

READING=one-way SHIM_MAX_STATE_TOKENS=126976 SHIM_SELECTOR=none bash h2o-lightning-31b/serve.sh --profile jevbench

This starts vLLM on port 8000 and the shim on port 8741:

setting (stock vLLM, the Decision Index command)	value
attention	one-way (stock vLLM generate runner; the default)
weights	FP8 (vLLM's online FP8 of the bf16 checkpoint, `--quantization fp8`; the default)
context (`MAX_LEN`)	131,072 tokens (the default)
readout orders	one read (`1`, the default)
record (`state`)	read whole, up to 126,976 tokens; a record that does not fit is refused with HTTP 422, never cut
Box questions	on wherever images are read (no setting); needs an engine with structured outputs

The Decision Index kit does not allow an engine to shorten its inputs; a refused record is scored as a wrong answer.

Default mode. --profile jevbench sets no shim profile: it makes a request that names no mode use the server's own settings, i.e. exactly the two settings above (one read, every record whole). It is not the JevBench runbook (that is run_default.sh, on our build). Plain serve.sh (no settings) is the AA setup on our build (see Default configuration (AA mode)).

Timeouts. The shim waits up to 180 s for vLLM's answer to a request (SHIM_TIMEOUT, in seconds), sends it once more on a fresh connection, and otherwise answers HTTP 502, which the kit scores as a wrong answer (and three failures in a row can stop a kit run). At start-up the shim prints shim: NOT the benchmarked configuration: SHIM_SELECTOR, SHIM_MAX_STATE_TOKENS override serve_config.json. That line is expected with this command: serve_config.json holds the serving default (3,000-token selection), and the two settings above replace it with the Decision Index configuration. For every other use, see API serving on stock vLLM.

3. Check that the server is ready and configured as above. curl -s http://127.0.0.1:8741/health must return "ok": true, "default_mode": "jevbench", "reading": "one-way (READING=one-way)", "max_state_tokens": 126976 and "selector": "none", and, under "readout", "orders": "off" (one read: "orders": 1 in serve_config.json is reported as "off") and "choice_cutoff": 0.8 (reported even while the second read is off).

Then run the kit as in step 3 of the Decision Index section.

API serving on stock vLLM: the jevbench mode (3,000-token selection)

For general API serving on stock vLLM, not for either benchmark: READING=one-way bash h2o-lightning-31b/serve.sh --profile jevbench with no other settings. It is the one-way fallback above, except for long records. A state of up to 3,000 tokens is sent whole. From a longer one, the shim keeps the parts most relevant to the question: whole paragraphs, lines or fields, ranked by BM25 against the question and its options, with their headings, up to 3,000 tokens. The question and its options are never cut. When no part of a long record matches the question, or the record cannot be split into fitting parts, it is cut to its head and tail around a ... line and answered (effort_used.state_selection "no match; head+tail"); selection never refuses. This keeps every request short and its latency even.

Not re-measured on this release: how much the 3,000-token selection costs against reading long records whole.

Inference options

For the Decision Index, use only the runbook block above; nothing in this section changes it. The same holds for JevBench and its block. This section lists every option the model has, so that other uses (your own API serving, other benchmarks) can choose; for each it says whether it is on in the two benchmark runbooks (where the profile sets it, and you set nothing) and what it is for elsewhere. "Use it for" never means "add it to a benchmark runbook".

Three setups have full benchmark numbers on this card, and every number below names the one it comes from (a few Measurements of the options. The option effects quoted on earlier cards were measured on earlier weights and are not repeated here; on this release only the runbooks' own numbers above are measured. Each option below says what it does and where it is on.

Two-way reading

What it does: the full-attention layers read the prompt both ways (the reading the model was trained for), per request, on our vLLM build.
Decision Index runbook: on. JevBench runbook: on. Set by the profiles.
Elsewhere: needs our vLLM build (plain serve.sh reads two-way by default, or serve.sh --profile two-way); stock vLLM reads one-way only (READING=one-way). Use it for: any serving on our build.

Adaptive reasoning

What it does: for a request in the aa mode, the one-pass read also returns the model's own residual activations at the last prompt token (two layers); a small head decides from them, in about a millisecond, whether the request would gain from deliberation, and if so the same weights on the same engine think about it (a budget of 2,560 thinking tokens, the same on any load or hardware; at most 4 such passes per engine at once with further requests waiting their turn: an escalated request is never shortened, skipped or replaced because of load), the answer is read from the analysis and returned; the analysis is never returned. Single-question choice requests without images only, with a one-pass prompt of at most 8,192 tokens (a longer one keeps the single pass, effort_used.router.skipped: "too_long").
Decision Index runbook: off (decision_index requests never escalate, and the profile starts without it). JevBench runbook: off (jevbench requests never escalate, and the profile starts without it).
Elsewhere: on by default for aa requests on a plain serve.sh start on our vLLM build (ADAPTIVE_REASONING=0 turns it off; under a --profile it is off unless ADAPTIVE_REASONING=1; LIGHTNING_ROUTER_MODES names the modes that may escalate, default aa). The hosted H2O-Lightning-31B service will run it with these defaults after its next update. Use it for: questions where a wrong answer costs more than a few seconds of thinking. Measured in Option details.

Routing long or heavy requests one-way

What it does: a record longer than the two-way window, or a request whose questions add up to more than the latency cap of two-way prompt tokens, is read one-way, whole.
Decision Index runbook: on (cap 4,096 summed two-way prompt tokens; window 8,192). JevBench runbook: on (the same). Set by the profiles. A request is read two-way only when its questions' prompts add up to at most the cap.
Elsewhere: SHIM_AUTO_MAX_TOTAL_TOKENS sets the cap. Use it for: keeping two-way latency even.

Two-order reading (a second read of an unsure choice question)

What it does: a choice question whose first read is unsure (top probability under 0.8) is read again with its options reversed, and the two reads are averaged.
Decision Index runbook: off. JevBench runbook: off. (The Decision Index runbook's conditional re-read below is a different second read.)
Elsewhere: off by default; SHIM_READOUT_ORDERS=auto turns it on, for your own serving on stock vLLM one-way (not measured on this release).

Third and fourth reads

What it does: a choice question still unsure after two reads is read in rotated option orders (or once in the other reading) and all reads are averaged.
Decision Index runbook: off. JevBench runbook: off.
Elsewhere: no switch; not offered.

Conditional re-read (a second read of an unsure text answer)

What it does: a text choice or yes/no answer whose top probability is under a threshold is read a second time, with the request written out twice, and the two reads are averaged. Only requests of up to 2,000 tokens are re-read. Score questions are never re-read; images get the image re-read instead.
Decision Index runbook: on, below 0.90 (the decision-index profile, also accepted as balanced). JevBench runbook: off (the two-way profile reads once).
Elsewhere: serve.sh --profile decision-index-max re-reads below 0.95: more re-reads; not measured on this release. Use the re-read for: accuracy on unsure text answers when a higher median latency is acceptable.
Attribution: The re-read writes the request in FlexDM v1.0's header layout (FlexDM, MIT; the layout is Will Cole's, MIT), with decider's chat-layout parts (Apache-2.0) and shisa-ai/shisa-de-2's read-twice; the notices are in NOTICE.

Image re-read at a larger image budget

What it does: an image choice or score question whose top probability is under 0.8 is read again at Gemma-4's 1120-token image budget (instead of 560) and the two reads are averaged.
Decision Index runbook: on (1120, under 0.8). JevBench runbook: off. Set by the profiles.
Elsewhere: off by default (SHIM_IMAGE_REREAD_BUDGET=1120 SHIM_IMAGE_REREAD_BELOW=0.8 turns it on). Use it for: accuracy on images when latency matters less; it about doubles the 95th-percentile latency of an image request, which is why the JevBench runbook (whose speed score reads the 95th percentile) leaves it off.

Per-type temperatures and the yes/no floor

What it does: each answer's probabilities are tempered by question type (fitted on v1.3.0's weights from Decision Index 0.3 public per-row reads; see Calibration), and a yes/no answer is reported at no less than 0.801 on its chosen side (the answer itself never changes).
Decision Index runbook: on. JevBench runbook: on. Stock vLLM one-way: on. The shipped values: one-way choice 1.28, score 0.7, yes/no 0.6, images 0.9; two-way 1.13, 0.7, 1.07, images 0.9; floor 0.801. The decision-index profiles read images at 0.96 in both readings (see Calibration).
Elsewhere: on as shipped; SHIM_TEMPERATURE*, SHIM_IMAGE_TEMPERATURE and SHIM_NOUL_FLOOR (0 turns the floor off) change them, for example behind a fallback that needs raw probabilities. See Other settings.

Long records: read whole, or cut down by BM25

What it does: a record over the token budget is either refused (read whole, HTTP 422) or cut down to its most relevant parts by BM25 selection (with no match, its head and tail are kept; never refused).
Decision Index runbook: read whole up to 126,976 tokens, refused above (the kit forbids shortening inputs, and its longest record fits). JevBench runbook: whole up to 126,976 tokens, BM25 above.
Stock vLLM one-way, Decision Index command: read whole up to 126,976 tokens. Plain serve.sh (aa mode): read whole, as the Decision Index runbook. The jevbench mode on plain serve.sh: BM25 down to 3,000 tokens. Use it for: general API serving, where short prompts keep latency even.

Weights: FP8 or bf16

What it does: vLLM's online FP8 of the bf16 checkpoint, or the bf16 checkpoint itself.
Decision Index runbook: FP8. JevBench runbook: FP8. Stock vLLM one-way: FP8 (the default).
Elsewhere: QUANT=none serves bf16. Use FP8 on one GPU: bf16 does not fit the Decision Index's long records on one 80 GB H100. See FP8 or bf16 weights.

Bounding-box questions

What it does: a box question answers with a bounding box in an image.
Decision Index runbook: on. JevBench runbook: on. Everywhere else: on. There is no setting to turn it off: a box question on an image deployment is answered by one guided (JSON-schema) generation. An engine without structured outputs refuses it with HTTP 422; requests without a box question are unaffected. Not measured on the Decision Index. See Box questions.

Structured-output backend: llguidance (opt-in)

What it does: GRAMMAR_BACKEND=guidance bash serve.sh starts vLLM with llguidance (no free whitespace) instead of vLLM's default xgrammar, for the letter mask of every read of up to 26 options and the Box generation.
Decision Index runbook: off. JevBench runbook: off. Not available with --profile (a profile fixes the engine).
Elsewhere: off by default. The readout and the Box confidence map were fitted on xgrammar.

FlashAttention 4 on Hopper (opt-in)

What it does: ATTENTION_BACKEND=fa4 bash serve.sh runs the two-way reads on FlashAttention 4 instead of the Triton backend, on Hopper GPUs (H100, H200) only.
Decision Index runbook: off. JevBench runbook: off. Not available with --profile.

Read timeout and the start-up checks

What it does: a vLLM answer slower than 180 s is sent once more, then answered HTTP 502; before its first answer the shim checks that the engine reads both ways (see Two-way start-up check).
Decision Index runbook: on. JevBench runbook: on. Stock vLLM one-way: the timeout is on (SHIM_TIMEOUT).
Start-up: when the shim refuses a running engine for good (the chat rendering, the context, the readouts, a deterministic engine whose one-way reading sees later tokens), serve.sh stops at once and prints the shim's reason.

Option details

The measurements behind the options above. None of them changes a benchmark runbook.

Adaptive reasoning (on for aa requests)

The head (router_head.npz) reads the residual stream of two layers at the last prompt token and predicts, for the one-pass answer and for the thinking answer, the probability of being right; a request is escalated when the predicted gain clears the served threshold. Every number below was read on this release's files with the served settings (the 2,560-token thinking budget); the accuracy rows compare each request's served answer with its own one-pass answer (paired bootstrap, 2,000 draws); the head itself was fitted on the activations of v1.3.0's weights, on the GPQA-diamond, SuperGPQA and control-set items below and on a 592-item mixed set; 200 of the 513 HLE items were in the head's training set, in a rendering with the options permuted. The held-out reads are the 313 HLE items the head never saw and JevBench's 231 items. For the sets the head was fitted on, the table applies the decisions of a head fitted on the other half of the items (two folds by item: the sha256 halves where the items carry them, random halves for the control-set items) to the measured answers; an item that only the held-out head would escalate keeps its one-pass answer there, so those rows are lower bounds, except the control-set row, which was read a second time with every request sent to the thinking pass so that its held-out number is measured in full; the in-sample reads are given beside them, labelled.

	value
HLE-like requests sent to the thinking pass (HLE-text-MC, the 313 held-out items)	83.1%
accuracy gain on HLE-text-MC, held out (313 items, paired bootstrap 95% CI)	+10.86 points [+6.39, +15.02]
the other 200 HLE items (seen in the head's training in a permuted-options rendering; 82.5% escalated)	+11.50 [+5.50, +17.50]
decision-style control set (310 decision-style questions drawn from ANLI, ARC, HellaSwag, MMLU, MuSR, WinoGrande and other benchmarks, read in `aa` mode end to end; one-pass accuracy already 84.2%): held-out (random-half) decisions, 13.2% escalated, measured	gain not established: +0.32 [-1.61, +2.26] with all 41 held-out escalations measured (a second read of the set with every request sent to the thinking pass); in-sample, the shipped head's own 30 escalations (9.7%, fitted on these items): +3.23 [+1.29, +5.48] on the first read, +2.58 [+0.65, +4.52] on the second, 26 of the 30 thinking answers repeated
GPQA-diamond (198 items) / SuperGPQA (300): held-out-half decisions, 90.9% / 45.7% escalated, lower bounds	+18.69 [+11.11, +26.26] / +0.67 [-2.67, +4.00] (10 of 180 and 23 of 137 held-out escalations have no measured thinking answer); in-sample: 87.9% escalated, +18.18 [+10.61, +25.76] / 51.0%, +0.33 [-3.33, +3.67]
over the 1,164 thinking passes of the 2,560-budget read of HLE, GPQA-diamond, SuperGPQA, the control set and the mixed set (read 16 passes at a time, four times the served limit): cut by the 155 s safety net / answered busy / stopped at the token budget	0 / 0 / 678
decision stability of the head: repeated reads of 40 HLE rows (warm cache) / against the head's training-time activations (200 HLE rows)	100% / 97.5% identical decisions
customer-like requests escalated (JevBench's 231 public items in `aa` mode, one at a time)	22 of 231 (9.5%)
of those, thinking passes that finished / reached the token budget / were cut by the safety net / found no slot	16 / 6 / 0 / 0
wall time of an escalated request at 16 in flight (4 thinking passes at a time, the rest waiting; p50 / p95)	191.9 s / 289.3 s (plain requests 112 ms / 1.18 s; the acceptance client)
at 16 in flight, the acceptance client's request form of the same 231 items (it escalates 40 of them; the harness's own rendering of the items above escalates 22: the head reads the activations of the request as rendered): escalations shed (none by default: every selected request gets its thinking pass), budget and safety-net cuts, and the answers against one at a time	0 shed; budget cuts 9 vs 13, safety-net cuts 0 vs 0 (of 40 vs 40 escalations; 0 busy answers); 1 of 231 answers differ from one at a time, against 0 for a one-at-a-time repeat of the 40 escalated items
plain reads at 16 in flight while escalations run (p50 / p95), and idle	1083 ms / 1.5 s during the escalations, 989 ms / 1.3 s idle (160 one-way reads at 16 in flight)

The control set checks that the router does not spend thinking where one pass already does well: first-pass accuracy there is 84.2%.

How it runs. The engine is our vLLM fork, h2oai/vllm at commit a36449a18b4465b4f70fa078f5403032de414de8 (the per-request two-way attention build plus the residual-activation return), laid over the matching stock wheel; run_decision_index.sh installs it and serve.sh checks it (engine/vllm_per_request_attention.SHA256SUMS). The thinking pass uses the same weights on the same engine. It stops at a TOKEN budget: 2,560 thinking tokens (LIGHTNING_REASON_BUDGET), so the cut point is the same on any load or hardware; a pass that reaches it is re-read from its partial analysis (effort_used.reasoned.finish: "length"). At most 4 passes run per engine at once (LIGHTNING_REASON_MAX_INFLIGHT); a further request waits for a free slot as long as it takes, recorded as effort_used.reasoned.queued_s: an escalated request is never shortened, skipped or replaced because of load, it waits instead. In our tests the escalation decision was identical on every repeat; the thinking pass is sampled (temperature 1), so an escalated answer can differ between runs, under load or not. Two bounds never produce a single-pass answer: the engine must start streaming the thinking within LIGHTNING_REASON_TTFT_S (10 s), else the request is answered HTTP 503 (busy, retry), and a wall-clock safety net (LIGHTNING_REASON_DEADLINE_S, 155 s counted from the first thinking token, never from the wait) sits above the budget's worst-case time and is not expected to fire (status "deadline" when it does). A one-pass prompt longer than LIGHTNING_REASON_MAX_PROMPT_TOKENS (8,192, the two-way window) is never escalated (router.skipped: "too_long"). Clients: in the default aa mode a decision may run a thinking pass before it answers (the pass stops at the 2,560-token budget; the 155 s safety net, counted from its first thinking token, is the longest it can take), and nothing is sent until the answer: use a read timeout of at least 180 s, as serve.sh states; under load a decision also waits for one of the 4 thinking slots (it is never shortened or answered without its pass), so a client sending many requests at once should allow more, see the 16-in-flight row above; decision_index and jevbench never reason. The compile cache of a start with adaptive reasoning on is kept apart from one without it (serve.sh suffixes -ar30_40 to the cache folder, also to an explicit VLLM_CACHE_ROOT). Load shedding is an opt-in that changes answers under load and is off by default: LIGHTNING_REASON_SHED_AFTER_S=<seconds> makes a request that waited that long for a slot keep its one-pass answer (effort_used.reasoned.status: "capped"). The hosted service will run these values after its next update. Per request only "reason": {"enabled": false} may be sent; any other reason.* key is refused with HTTP 400, so a caller cannot raise a deployment's cost settings. ADAPTIVE_REASONING=0 turns it off for the server: the engine then returns no activations and every request is the single pass. It is also off, with a warning, when the server reads one-way (READING=one-way, or the one-way fallback without our build), and off under a --profile (the runbooks) unless ADAPTIVE_REASONING=1 is set.

Two-way reading (on in both runbooks; needs our vLLM build)

The model was trained to read with its full-attention layers looking both ways (two-way, or bidirectional, attention) as well as one-way. Both benchmark runbooks read two-way by default.

How: our vLLM build (per-request attention) serves both readings, chosen per request. The shim marks each two-way read (vllm_xargs attention_mode: bidirectional); the engine prefills that prompt whole with the full-attention layers reading both ways and the sliding layers one-way, and keeps it out of the prefix cache. An unmarked read is stock one-way. Before its first answer the shim checks that the engine reads both ways: see Two-way start-up check.
Routing: a request reads two-way unless its record is longer than the two-way window, its summed two-way prompt is over the latency cap, or it has a Box question; those read one-way, whole. A request can choose with "kwargs": {"attention_mode": "two-way"} or "one-way".
Cost: a two-way read computes its whole prompt (no prefix cache), so a request with many questions about one long record costs more two-way than one-way; the latency cap sends those one-way.
Parity: the per-request attention was checked in bf16 on GPU: where the one-way and two-way readings give different clear answers, an unmarked read gives the one-way answer and a marked read the two-way answer.

Two-way reading on other GPUs

The Decision Index and JevBench runbooks do no GPU detection: they assume one RTX PRO 6000 and refuse a smaller GPU. Only plain serve.sh (two-way by default: your own serving on our build, without a profile) looks its sizes up in the two-row table below. It reads the GPU's memory at start and picks the engine's sizes itself; only measured GPU classes have sizes, and on any other GPU it stops and asks for them (or for READING=one-way). Set values always win.

GPU class (memory reported by `nvidia-smi`)	`MAX_LEN` (context)	`TWO_WAY_WINDOW`	`GPU_UTIL`
96 GB (92,000 to 100,000 MiB; one RTX PRO 6000 96 GB)	131,072	8,192	0.92
80 GB (76,000 to 86,000 MiB; one H100 80GB)	81,920	16,384	0.92

Other GPUs (e.g. H100 80 GB): these sizes were not run with this release, and are not the Decision Index setup. The Decision Index configuration above is one RTX PRO 6000 96 GB only.

The engine's prefill step (--max-num-batched-tokens) is the two-way window plus one token for each running sequence (MAX_NUM_SEQS, default 1024, vLLM's own default on these GPUs), so a window-sized two-way prompt runs beside a full batch of running answers instead of waiting for them to finish; the shim still reads nothing longer than the window two-way.

On another GPU, set MAX_LEN, TWO_WAY_WINDOW and GPU_UTIL (the same three sizes as for the runbooks). PLAN_SIZES=1 bash serve.sh prints the sizes it would use and starts nothing. The Decision Index runbook sets the 96 GB values itself.

FlashAttention 4 on Hopper (opt-in). ATTENTION_BACKEND=fa4 bash serve.sh runs the engine's two-way reads on FlashAttention 4 instead of the Triton backend; it works on Hopper GPUs (H100, H200) only, and our build refuses a two-way read on any other GPU, so the start-up check stops the script there. Not available with --profile (a profile fixes the engine). Not measured on this release.

Long records

The two benchmark runbooks differ in two places: here, and the 1120 image re-read (on in the Decision Index runbook only). SHIM_MAX_STATE_TOKENS=<n> sets the budget (it never exceeds MAX_LEN - 4,096); SHIM_SELECTOR=none sends every record whole and answers 422 when one is over the budget; SHIM_SELECTOR=basic is a simpler paragraph selector.
Read whole with refusal (SHIM_SELECTOR=none) is used only by the Decision Index runbook and the Decision Index fallback command, because the kit does not allow shortening inputs and its longest record fits the context. Every other setup selects an over-context record down to fit with BM25 and never refuses it (a record with no part matching the question, or one the selector cannot cut into fitting parts, is cut to its head and tail around a ... line, as in training, and answered): the default command (run_default.sh, up to 126,976 tokens whole) and the jevbench mode of plain serve.sh (3,000-token selection), on either GPU class.

FP8 or bf16 weights

Default: FP8 (vLLM's online FP8, W8A8 dynamic, about 32 GiB of weights on the GPU).
Switch: QUANT=none bash serve.sh serves bf16 (about 59 GiB of weights plus the KV cache; lower MAX_LEN to fit).

Image re-read (on in the Decision Index runbook)

An image choice or score question whose top probability is under a threshold is read again at a larger image budget (Gemma-4's 1120 soft tokens instead of the 560 the shim sends), and the answer is the mean of the two reads (yes/no image questions are read once).

Default: off (SHIM_IMAGE_REREAD_BUDGET 0). The Decision Index runbook turns it on at 1120 under 0.8, with SHIM_IMAGE_ORDERS=1 (no option-order second read for images, so the 1120 re-read is the only extra image read). The default command and the API setup keep it off: it about doubles the 95th-percentile latency of an image request, and JevBench's speed score reads the 95th percentile.
Switch: SHIM_IMAGE_REREAD_BUDGET=1120 SHIM_IMAGE_REREAD_BELOW=0.8; /health reports the setting and counts the re-reads (auto.image_rereads with the two-way reading; images.reread_count on a one-way server).

Box questions

A box question ({"type": "box", "instructions": "the coffee mug in photo"}) answers with a bounding box in fractions of the image, its confidence, and the probability that the target is present. On by default: in normal serving and in both benchmark runbooks (run_decision_index.sh, run_default.sh), wherever images are read (there is no on/off setting). The box comes from one guided (JSON-schema) generation, so the engine needs structured outputs (vLLM has them on by default); an engine without them refuses a request with a box question with HTTP 422, and requests without one are unaffected. A text-only deployment refuses a box question with HTTP 400. box.mode "generate" (one constrained generation of the box) is the mode to use with this release.

ImageJevBench Box track: how to run

Ask a Box question through the package, not through a chat prompt. On a stock vLLM chat endpoint the weights answer with a box but are not trained to state a confidence for it: in this model the box's confidence is computed by the shim, from the model's own yes/no readout of the box drawn on the image ("does the red box tightly contain the target?"). Run the server, one request at a time on one GPU:

hf download h2oai/h2o-lightning-31b --revision v1.3.4 --local-dir h2o-lightning-31b
bash h2o-lightning-31b/run_default.sh        # prints DEFAULT SERVER READY on http://127.0.0.1:8741

One request per item, the image in the record and one box question that names it:

curl -s http://127.0.0.1:8741/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"photo": "data:image/jpeg;base64,<the image, unchanged>"},
  "questions": {"where": {"type": "box", "instructions": "the leftmost dog in `photo`"}}}'

Read answers.where.box (left, top, right, bottom, fractions of the image from the top-left corner) and answers.where.confidence (the readout's estimate that the box overlaps the target with IoU >= 0.5). If answers.where.found is false, the box is the whole image and the confidence is 0. present is whether the named object is visible; it is not the box's score.

The box confidence was checked on v1.3.0's weights on an out-of-distribution box set (RefCOCOg validation, 600 questions, board images excluded): calibration error 0.041 with the readout's own probability, accuracy at IoU 0.5 87.2 %; a fitted map did worse held out both ways, so none is applied. We have not run this model on the Box track.

Other settings

SHIM_NOUL_FLOOR=0 turns the yes/no commit floor off, so P(yes) is reported as read (for a deployment behind a fallback). As shipped, a yes/no answer is reported at no less than 0.801 on its chosen side; the answer never changes.
The calibration temperatures, fitted on v1.3.0's weights (see Calibration): one-way (serve_config.json) choice 1.28, score 0.7, yes/no 0.6, images 0.9. Two-way (serve_config_two_way.json) choice 1.13, score 0.7, yes/no 1.07, images 0.9; the decision-index profiles read images at 0.96. SHIM_TEMPERATURE (every type), SHIM_TEMPERATURE_CHOICE / _SCORE / _NOUL and SHIM_IMAGE_TEMPERATURE override them.
serve.sh keeps vLLM's compile cache in one folder per reading mode, image setting and weight format (~/.cache/vllm/h2o-lightning-31b/<mode>-<images>-<weights>): the engine would otherwise reuse a graph compiled for another image setting (for example MAX_IMAGES=0) and fail at start-up. VLLM_CACHE_ROOT overrides it.
One-way (READING=one-way, no profile), serve.sh also starts vLLM with --mm-processor-cache-gb 0 (vLLM's front-end image-processor cache off: on our hosted engines it went out of step with the engine's own cache on the first image after a start and answered HTTP 500; answers do not depend on it) and --allowed-media-domains media.invalid (vLLM never fetches a URL; the shim sends images as data). The two-way server, with or without a profile, keeps exactly the engine flags of the measured benchmark runs.
MAX_LEN, GPU_UTIL, VLLM_PORT, PORT and VLLM_ARGS adapt serve.sh to other hardware.

The API

Send requests to POST /v1/systemone. One request can carry several questions about the same state:

{"state": "Ticket 4471: since 09:10 the checkout page returns HTTP 500 for every customer; no orders have gone through in 40 minutes. Severity guide: P1 = a revenue path fully down; P2 = degraded but working; P3 = cosmetic.",
 "questions": {
   "priority":  {"type": "choice", "instructions": "Which priority does the severity guide assign?",
                 "criteria": {"p1": "Revenue path fully down", "p2": "Degraded but working", "p3": "Cosmetic"}},
   "all_users": {"type": "noul", "instructions": "The problem affects every customer.",
                 "criteria": {"true": "Affects everyone", "false": "Affects only some"}},
   "urgency":   {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["Low", "Medium", "High"]}}}

Each answer carries its probabilities and a confidence. usage.input_tokens counts the prompt head that the questions share once; usage.output_tokens is one label token read per answered question.

Images. Put each image in the request as a data:image/...;base64,... URI, anywhere inside state or in a top-level images list. Up to 4 images per request; image answers use their own temperature (serve_config.json, "image"). The shim also serves POST /v1/decisions (OpenAI's Decisions API shape, client.decisions.create in the openai Python package).

Status codes. An input the shim cannot take (more than 255 options, an unknown question type, a malformed question, more than 4 images, an image vLLM cannot load) gets HTTP 422; an empty request gets 400. A misconfigured vLLM (wrong served name, too small a context, a different chat rendering or tokenizer) is a 503, so a broken server stops a run instead of scoring it. An optional extra read (the text re-read, the image re-read, the reverse-order second read) that cannot be made never fails a request: the first answer stands, and effort_used and /health say so. On a server whose two-way start-up check did not pass, an explicit two-way request gets HTTP 400.

Tests (no GPU): python3 h2o-lightning-31b/test_shim.py and python3 h2o-lightning-31b/test_security.py.

Limitations

Evaluated mostly in English. Up to 255 options per question; a request with more gets 422.
The probabilities are calibrated for decision questions of this kind, not for open-ended text.
Not a chat model: it answers one decision per question and does not generate explanations.
No video input.
The engine's compile and kernel-tuning cache (VLLM_CACHE_ROOT) can differ between installs and shift probabilities slightly, occasionally flipping a near-tie answer; reusing the same cache reproduces results exactly (two fresh compiles, read on the same 203 vision rows, differed in 7 top answers).
The "I don't know" cutoff (aa mode) was fitted on our question mix; on much harder questions a higher cutoff scores better (see Modes).

Credits

Cygnet (blockbrain-ai/cygnet-recipe, commit 74f9555, MIT): the prompt layout and the one-token option-letter readout this shim implements. Its MIT copyright and permission notice are in LICENSE-cygnet. Cygnet credits the one-token readout approach to NInfer.
Header layout (the conditional re-read of the decision-index and decision-index-max profiles, and SHIM_LAYOUT=header): built token for token as FlexDM v1.0 builds it (MIT), a layout FlexDM takes from Will Cole (MIT); their MIT notices are in LICENSE-flexdm and LICENSE-wcole3. Its chat layout parts and the _index position annotation of long arrays are decider's (Apache-2.0). Reading the user turn twice follows shisa-ai/shisa-de-2.
Weights: built on Google DeepMind's gemma-4-31B-it (revision 842da3794eaa0b77d5f08bae87a17459d91ff475, Apache-2.0).
Serving: vLLM 0.30.0 (Apache-2.0), unmodified for the one-way fallback; the two-way reading runs on our vLLM build with per-request attention (h2oai/vllm, a modified vLLM 0.30.0, Apache-2.0). This repository ships no vLLM code; the Apache License 2.0 is in LICENSE-APACHE-2.0.

See NOTICE.

License

Non-commercial. See LICENSE.

Based on google/gemma-4-31B-it (Apache-2.0, Copyright Google); Apache-2.0 text: https://www.apache.org/licenses/LICENSE-2.0

The portions of the shim taken from Cygnet are under the MIT License in LICENSE-cygnet. The header layout's portions taken from FlexDM and Will Cole are under the MIT License in LICENSE-flexdm and LICENSE-wcole3; decider's under the Apache License 2.0 in LICENSE-APACHE-2.0.