1000luca/FlexDM-LoRA-v2.1

Leaderboard · Public run · Submission (PR #367) · FlexDM reader · All versions · Blog

FlexDM-LoRA ver 2.1 · Gemma-4-31B

A 4 GB LoRA adapter that turns stock Gemma-4-31B-it into a calibrated decision model. 4th of 275 on the Decision Index, with the second-highest new-domain score on the board.

Give it a state (a document, a conversation, a device list, a JSON record, ...) and one or more multiple-choice or yes/no questions about it. It returns a probability for every option, read from the option-label logits at Answer: ( in a single prefill. It generates no text. It is served exactly like FlexDM (same prompts, same temperatures, same engine), with this adapter loaded in vLLM on top of the FP8 base model.

Highlights

• 4th of 275 on the Decision Index (edition 0.4.1, board data of 2026-11-06), with a Full score of 71.61. The board counts scores within 0.9 points as tied and lists it in the #2 group, with KnowLine 31B Gen1.5 (71.91) and Sieve 31B v3 (71.10).
• New domains 79.92, 2nd of 275. Only REAX 1.3 (81.38) is higher, an ensemble of two parts that are each blended from two fine-tuned models. It is the highest new-domain score of any single model, and of all 84 LoRA and LoRA + head entries.
• +4.05 Full from the adapter alone. FlexDM, the same reader on the untrained base model, scores 67.56. The adapter adds +3.41 on the private same-skills tests and +5.00 on the private new-domain tasks, and raises 24 of the 34 private same-skills tasks (see Where the adapter helps).
• Light to ship. One 4 GB adapter file on the unchanged base model, FP8 at load, one GPU. The board measured median 155 ms and mean 416 ms per request on one RTX PRO 6000.
• Calibrated probabilities. On the board's calibration measurement: accuracy 77.9%, mean confidence 80.9%, ECE 0.054, Brier 0.305 (FlexDM: 75.3%, 78.8%, 0.044, 0.355).
• Fully disclosed recipe. The training data, loss, selection rule and decontamination are described below. The only teacher is the frozen base model itself. No Jev outputs and no outputs of other entries were used.

Leaderboard

The Decision Index leaderboard (Full score view, board data of 2026-11-06), with the FlexDM-LoRA 2.1 bar selected: Full 71.6, rank #2 (tied), median latency 155 ms.

Top of the board (edition 0.4.1, board data of 2026-11-06 09:05 UTC, 275 entries). The Full score is 0.6 × private same-skills tests + 0.4 × private new-domain tasks, both on the public scale, as the board shows them. The public score is shown on the board but does not count.

#	Model	Board type	Same skills	New domains	Full
1	REAX 1.3	inference technique (two-part ensemble)	69.04	81.38	73.98
2	GLiDE 27B	full fine-tune	69.83	78.68	73.37
3	KnowLine 31B Gen1.5	LoRA	66.79	79.59	71.91
4	FlexDM-LoRA 2.1 · Gemma-4-31B (this model)	LoRA	66.07	79.92	71.61
5	Sieve 31B v3	full fine-tune	68.27	75.33	71.10
6	Turion Jev 31B	LoRA	65.04	78.60	70.47
7	vLLM-SR d3-ultra (27B)	full fine-tune	66.49	75.71	70.18
8	Bespoke Nimble v4.1 Max (27B)	full fine-tune	67.12	74.39	70.03
22	FlexDM · Gemma-4-31B (same reader, no training)	inference technique	62.66	74.92	67.56

Model summary

Base model	google/gemma-4-31B-it @ 842da3794eaa0b77d5f08bae87a17459d91ff475 (Apache-2.0), unchanged, quantized to FP8 per block at load
Adapter	LoRA r 128 = the exact average of two r 64 training runs, applied at strength 0.7 (lora_alpha 89.6), on q/k/v/o/gate/up/down_proj of all 60 text layers; 4.0 GB
Reader	FlexDM: questions first, requests of at most 3,000 tokens read twice, low-confidence questions read again with the options reversed
Output	a probability for every option, from the option-label logits; no generation
Question types	multiple choice (2-255 options) and yes/no (noul); rubric score questions are refused as unsupported
Context	up to 131,071 tokens per row; longer rows are refused, never truncated
Serving	vLLM 0.30.0, unmerged LoRA, max_lora_rank 128, prefix caching, one GPU with about 75 GB free
Board class	LoRA

How it reads

Gemma is causal: a token only sees the tokens before it. If the state came first, the model would read it without knowing what will be asked. FlexDM puts the questions first and lets the model read short requests a second time; this adapter is trained on exactly that rendering.

<chat head>Questions that will be asked about the context:      1. every question of the request, first
- <question 1>
- <question 2>

Context:
<state>

Question: <question i>
Options:
(A) <option>
(B) <option>

Read the same input again before answering:                      2. a second copy, for requests of at most 3,000 tokens
<question list, state, question and options again>
<chat tail, thinking off>Answer: (                               3. read the option-label logits here

1. Questions first: the state is read with the questions already in view.
2. Read twice for requests of at most 3,000 tokens: every token of the second copy sees the whole first copy, options included. Longer requests are read once.
3. Label readout. vLLM runs one step and returns the log-probabilities of the option labels. Probabilities are a softmax at a per-type temperature: choice T = 10.9757 - 1.7016 ln(n options), yes/no T = 10.732.
4. Second look when unsure. A question whose top option has probability below 0.6 is read once more with its options in reverse order, and the two distributions are averaged (reverse_below=0.6).

All rows of a request share one cached prefix (chat head, question list, state), so a request with many questions pays for its state once. Every benchmark uses this same rendering: no per-benchmark prompts, rules or post-processing.

Quickstart

Requirements: Linux x86_64, Python 3.12, an NVIDIA driver for CUDA 13.0 or newer, one GPU with about 75 GB free (we use an RTX PRO 6000 Blackwell 96 GB), and uv.

hf download 1000luca/FlexDM-LoRA-v2.1 --local-dir FlexDM-LoRA-v2.1
uv venv .venv --python 3.12 && source .venv/bin/activate          # .venv/bin first on PATH: FlashInfer's JIT calls its ninja
uv pip install -r FlexDM-LoRA-v2.1/code/requirements.txt --torch-backend=cu130   # the exact environment of the run, kit included
hf download google/gemma-4-31B-it --revision 842da3794eaa0b77d5f08bae87a17459d91ff475   # gated: accept the Gemma terms first

Run any Decision Index rows file (or the public 0.3 suite built with the kit) with the submitted settings:

export PYTHONPATH=$PWD/FlexDM-LoRA-v2.1/code HF_HUB_OFFLINE=1
python -m decision_index run --engine jevlab.vllm_lora_reader:VllmLoraReader \
  --option lora=$PWD/FlexDM-LoRA-v2.1 --option layout=header --option read_twice=true --option twice_max_tokens=3000 \
  --option temperature_choice=10.9757 --option temperature_choice_log_slope=-1.7016 \
  --option temperature_noul=10.732 --option noul_bias=0 --option reverse_below=0.6 \
  --rows my-rows.jsonl.gz --compact --out runs/flexdm-lora

• The engine reads max_lora_rank (128) from the adapter and sets VLLM_USE_DEEP_GEMM=0 itself (block FP8 with DeepGEMM fails on sm_120). It stops on an unknown option rather than ignoring it.
• vLLM settings: offline LLM, fp8_per_block quantization at load, TRITON_ATTN, prefix caching, max_model_len 131,072, gpu_memory_utilization 0.70, enable_lora, unmerged adapter. The first start quantizes the 62 GB bf16 checkpoint and compiles kernels (5-15 minutes); later starts take a few minutes.
• Load the root files in vLLM. For transformers / PEFT, use peft/ (see Files for why there are two copies).

Results

Decision Index 0.4.1 (the board)

	FlexDM (no training)	FlexDM-LoRA ver 2.1	Change
Full score	67.56 (22nd of 275)	71.61 (4th of 275)	+4.05
Private tests, same skills (60% of Full)	62.66	66.07	+3.41
Private tasks, new domains (40% of Full)	74.92	79.92 (2nd of 275)	+5.00
Public 0.3 index (shown, not counted)	63.63	68.99	+5.36
Calibration (board): accuracy / ECE / Brier	75.3% / 0.044 / 0.355	77.9% / 0.054 / 0.305
Latency (board, one RTX PRO 6000): median / mean / p95	109 / 301 / 1,106 ms	155 / 416 / 1,695 ms

Board data of 2026-11-06 09:05 UTC; 275 entries, including the Jev 1.14 reference.

Where the adapter helps

The board publishes each model's score on the private same-skills test of every task. Against FlexDM (the same reader without the adapter), ver 2.1 is higher on 24 of the 34 tasks, equal on 3 and lower on 7. The largest gains are on the task formats the training data was built for: device decisions, chord recognition, sarcasm, aspect-based sentiment, arithmetic and clinical-trial inference.

All 34 private same-skills tasks (board data of 2026-11-06)

Task	Area	FlexDM	ver 2.1	Change
Home appliances	Tools & Automation	60.8	84.0	+23.2
POP909	Arts & Human Taste	38.9	60.8	+21.9
iSarcasmEval	Language Understanding	61.1	72.0	+10.9
ACOS	Language Understanding	61.7	70.3	+8.7
GSM8K	Knowledge & Reasoning	82.5	88.9	+6.5
NLI4CT	Language Understanding	84.8	90.3	+5.5
GPQA Diamond	Knowledge & Reasoning	52.8	57.8	+5.0
API-Bank	Tools & Automation	87.8	91.9	+4.1
New Yorker	Arts & Human Taste	82.4	86.2	+3.8
VAST	Language Understanding	69.3	72.8	+3.5
CRUXEval	Knowledge & Reasoning	77.3	80.2	+2.9
MMLU-Pro	Knowledge & Reasoning	57.3	59.9	+2.5
BPoMP	Arts & Human Taste	95.7	98.2	+2.5
MuSR	Knowledge & Reasoning	65.8	68.2	+2.4
Amazon ESCI	Retrieval & Classification	68.8	71.1	+2.3
BANKING77	Retrieval & Classification	89.1	91.5	+2.3
BRIGHT	Retrieval & Classification	47.6	49.7	+2.1
CLadder	Knowledge & Reasoning	82.8	84.1	+1.3
HoVer	Retrieval & Classification	81.2	82.0	+0.8
ANLI	Language Understanding	69.0	69.5	+0.5
ChessBench	Knowledge & Reasoning	17.9	18.2	+0.3
WinoGrande	Knowledge & Reasoning	87.3	87.5	+0.2
CLINC150	Retrieval & Classification	82.0	82.2	+0.2
BBH	Knowledge & Reasoning	55.4	55.6	+0.1
BFCL	Tools & Automation	95.5	95.5	0.0
HellaSwag	Language Understanding	100.0	100.0	0.0
PhishNChips	Retrieval & Classification	92.3	92.3	0.0
When2Call	Tools & Automation	72.6	72.1	-0.5
RAGTruth	Language Understanding	71.6	71.0	-0.6
Habermas	Arts & Human Taste	49.6	48.6	-1.0
ToolRet	Tools & Automation	75.6	74.5	-1.1
Humicroedit	Arts & Human Taste	65.7	64.2	-1.5
FinEntity	Language Understanding	72.4	70.4	-2.0
SATA-Bench	Knowledge & Reasoning	46.4	38.3	-8.0

Scores are the board's private_eq values (equated to the public scale). Changes are computed before rounding.

Decision Index 0.3 public suite

Our run with the official kit (f8552a1) and scorer: 68.99 (raw 76.55), 140,178 of 140,178 scoreable requests answered, none unsupported, no errors. Results: 1000luca/FlexDM-LoRA-v2.1-decision-index.

Area (chance-corrected skill)	FlexDM	ver 2.1
Knowledge & Reasoning	51.1	58.0
Language Understanding	66.4	71.3
Retrieval & Classification	69.4	71.0
Tools & Automation	79.7	86.0
Arts & Human Taste	48.0	56.1
Public index	63.63	68.99

Latency of that run (RTX PRO 6000 Blackwell Workstation Edition, one request at a time, scoreable rows): median 174 ms, mean 493 ms, p80 780 ms, p95 2,136 ms. A separate process with the same adapter agreed on the top answer for 99.95% of 21,467 questions that the low-confidence re-read does not touch.

Versions

Version	Released	Training questions	Runs averaged	Served rank / strength	Low-confidence re-read	Public 0.3	Board Full
ver 2.1 (this model)	2026-11-04	180,101	2 (seeds 0, 1)	128 / 0.7	yes	68.99	71.61
ver 2.0	2026-11-04	115,249	1	64 / 0.5	yes	66.49	(replaced by 2.1)
ver 1.0	2026-10-25	90,007	1	64 / 0.5	no	65.47	(replaced by 2.1)
FlexDM (no training)	2026-10-20	none	none	none	no	63.63	67.56

The board lists the newest version of an entry, so ver 1.0 and ver 2.0 have no Full score of their own.

Training

• Runs. Two runs with the same data and settings, seeds 0 and 1 (different data order and option shuffles). Each run: PEFT LoRA r 64, lora_alpha 128 during training, dropout 0, targets q/k/v/o/gate/up/down_proj of the 60 text layers, float32 LoRA on bf16 weights; one epoch over 180,101 questions (110,456 requests), 1,524 steps of about 62k tokens (93.9M tokens), AdamW, lr 1e-4, 3% warmup then cosine decay, gradient clipping 1.0; four RTX PRO 6000 Blackwell (data parallel), about 14 hours.
• Trained as served. Rows are rendered exactly as served: the header layout, the 3,000-token read-twice rule, the option order shuffled per question, and one forward pass per request with the request's questions side by side.
• Loss per question, on the served distribution (softmax of the soft-capped label logits at the serving temperatures): cross-entropy to the gold label (or the source's annotator distribution) + 1.0 · KL(teacher ‖ model)
  • 0.3 · Brier (+ RPS for rater scales).
• The base model is the only teacher:
  • for most questions, Gemma read by FlexDM (vLLM FP8), averaged over the original and the reversed option order;
  • for the reasoning-heavy questions marked below, Gemma's own answer distribution after thinking (thinking mode on, 2 samples, up to 3,072-4,096 thinking tokens).
• Averaging. The two trained LoRAs (r 64 each) are stacked along the rank into one r 128 LoRA whose change is exactly their mean (ΔW = ½ ΔW₀ + ½ ΔW₁). The stored lora_alpha 89.6 applies 0.7 of it: W + 0.7 · ΔW.

Data

ver 2.0's data, minus 287 rows, plus about 65k new questions. Only sources whose licences allow training: no non-commercial or research-only sources, no Jev outputs, no outputs of other entries. Rows carrying a benchmark canary are dropped.

Part	Questions	What
ver 2.0's data	114,962	tasksource commercial-use rows; knowledge and reasoning train splits with thinking-mode labels; questions written by Gemma-4-31B-it itself; knowledge multiple choice; simulator-generated device decisions; POP909 chords from songs not in the suite; ver 1.0's rows; teacher-only rows. 287 rows with a negative gold answer were removed from five spam / toxicity sources (SMS Spam, hate_speech18, Wikipedia Detox, Jigsaw, Civil Comments), so their positive share matches the base model's instead of pulling every yes/no judgement toward "no".
Self-written product reviews	43,415	744 product reviews written by Gemma-4-31B-it itself, with aspect, category and opinion questions about each
Self-written emails	12,601	1,458 phishing and legitimate emails written by Gemma-4-31B-it itself from a written specification, with decision questions about each. Legitimate emails never carry shared-document links or come from personal webmail. 42 were dropped by decontamination.
Device decisions	4,838	300 more requests from the generator based on the kit's simulator logic (MIT), with a new seed. No state is equal to a public test row.
Clinical-trial statements	1,985	Fictional clinical-trial records generated by our code, with statements that are entailed or contradicted by a computed rule (50:50)
Causal questions	1,500	Random causal graphs generated by our code, with exactly computed answers (yes/no 50:50 per question type). The teacher is the base model's thinking-mode answer.
Next API call	800	Dialogues generated by our code over 53 API definitions taken from the API-Bank code (Apache-2.0), asking which API is called next
Official train splits of suite benchmarks	2,934 (1.6%)	Unchanged from ver 2.0: iSarcasmEval 1,500, GSM8K 1,039, Amazon ESCI 96, HoVer 95, BANKING77 56, WinoGrande 51, New Yorker (matching) 50, CLINC150 47. No test or evaluation split of any suite benchmark is used.

Decontamination. Every training row was checked against the 140,620 rows of the public 0.3 suite, the public JevBench items and jgeuter/OpenDecisions (exact strings, word 8-grams, long shared spans, a short-line fuzzy check, benchmark canaries). In the code-generated parts no record, causal graph or dialogue equals a public row; the device decisions share only the kit's fixed question and option wording.

Selection and disclosures

• Selection on public items. The candidates were seed 0 alone and the two-seed average, each at strength 0.5 and 0.7. The rule (the largest estimated gain, with floors on every task; the averaged and lower-strength candidate when within a margin) was written before the results were seen. It was applied to public Decision Index items (a 2,000-row stratified sample and fixed subsamples of the suite benchmarks), so the public score is not blind to the choice.
• Targeted data. The new generated and self-written parts were aimed at task formats where ver 2.0 scored low on public items. They were written from task descriptions; no public test row was used to write them.
• Held-out check only. jgeuter/OpenDecisions was read as a check and never trained on.

Limitations

• Tasks where it is lower than FlexDM. On the board's private tests, SATA-Bench (select all that apply) drops by 8.0 points, and FinEntity, Humicroedit, ToolRet, Habermas, RAGTruth and When2Call by 0.5-2.0 points. If your task looks like these, compare with FlexDM on your own data.
• Question types. Multiple choice and yes/no only. Rubric score questions are refused as unsupported.
• Text only. The engine loads the language model only (language_model_only); images are not read.
• Size and speed. The base model is dense (about 31B parameters in FP8, plus the adapter), so every token uses all of it. Requests of at most 3,000 tokens are written twice, and unsure questions are read again. Requests of 33,000-54,000 tokens take tens of seconds each.
• Precision. The submission runs the base in FP8 with the LoRA unmerged. Other precisions or a merged copy give slightly different probabilities; the board's reproduction bar for quantized entries is 96% top-answer agreement.
• Calibration. The temperatures were fitted for the untrained reader on our own off-suite data and kept unchanged for the adapter. If your use depends on probability thresholds, check calibration on your own data.
• Not measured. We did not measure robustness to instructions planted in the state. Keep untrusted text clearly delimited.
• Language. The training data is mostly English.

Files

Path	What
adapter_config.json, adapter_model.safetensors	The served adapter (load this one in vLLM). r 128, lora_alpha 89.6, all seven projections of the 60 text layers.
merge_info.json	How the served copy was made from the two trained adapters (the average, strength 0.7, the v_proj copies below), with their sha256.
peft/	The same adapter for transformers / PEFT.
code/jevlab/	The engine that ran the submission (sha256 as in the run's driver.json; the same files as ver 2.0).
code/requirements.txt	The exact Python 3.12 environment (vLLM 0.30.0, torch 2.13.0+cu130, transformers 5.16.1), kit included.
assets/	The banner and the leaderboard screenshot used on this page.

Why two copies. Gemma 4's full-attention layers have no v_proj (attention_k_eq_v: V is the k_proj output), so in transformers the k_proj LoRA changes both K and V. vLLM gives the V slice of its fused qkv_proj its own LoRA, so the served copy repeats the k_proj LoRA as v_proj on those 10 layers (5, 11, ..., 59). Without it, vLLM would serve V without the trained change. peft/ is the original for transformers.

Citation

@misc{flexdm_lora_v21,
  title  = {FlexDM-LoRA ver 2.1: a LoRA decision adapter for Gemma-4-31B},
  author = {1000luca},
  note   = {KAIST EE, FlexML Lab},
  year   = {2026},
  url    = {https://huggingface.co/1000luca/FlexDM-LoRA-v2.1}
}

License

• Adapter: CC BY-SA 4.0, because some training sources are CC BY-SA.
• Base model: google/gemma-4-31B-it, © Google, Apache-2.0 (Gemma 4 licence); not redistributed here.
• Code in code/: MIT (FlexDM).