license: apache-2.0
base_model: Qwen/Qwen3.5-4B
library_name: transformers
tags:
- decision-engine
- typed-decision
- choice
- classification
- merge
- qwen3.5
LemonDecider 4B
Model ID: lemondecider-4b-20261015-v2
Base model: Qwen/Qwen3.5-4B
LemonDecider 4B is a typed decision engine: it reads a state and a question with a list of options, runs one forward pass, and returns a probability distribution over the options (choice / noul questions) or an ordered rating (score questions). No text is generated and no input is ever truncated.
Model description
| Property | Value |
|---|---|
| Architecture | Qwen3.5-4B-class hybrid backbone (linear-attention + periodic full-attention) + a bias-free 255-way readout (LemonDeciderReadout, FP32 precision island), loaded through modeling_lemondecider.LemonDeciderForDecisionClassification (trust_remote_code) |
model_type / protocol |
lemondecider_decision / lemondecider-v1 |
| Attention | noncausal_full_attention — all-attend prompt for the periodic full-attention layers, causal recurrence for the Gated DeltaNet layers, SDPA only |
| Pooling / temperature | last-token pooling, temperature 1.0 (uncalibrated) |
| Max input length | 262,144 tokens (256K context) — the backbone's full position window; requests over the limit are rejected, never cut |
| Precision | backbone BF16, readout FP32 (readout GEMM executed in FP32 under autocast) |
| Serialized tensors | 724 (all vision tensors included; vision stem executed as a matrix-product patch projection) |
| Runtime | in-package Python runtime (modeling_lemondecider.py, configuration_lemondecider.py, contract.py, decision_utils.py, model.py), validated with transformers 5.17, flash-linear-attention 0.5.2, PyTorch SDPA |
Training
LemonDecider 4B was fine-tuned (SFT) from the common Qwen3.5-4B initialization on more than 700,000 public decision/QA rows collected by the project (about 805k train rows from ~818k collected rows incl. dev/test splits). Every source is public and permissive; no benchmark or evaluation data was used for training. The released weights combine two independent SFT runs over this corpus (weight-level combination, no training after that).
Question formats: choice (up to 255 options), noul (no/yes) and score (ordinal ratings); some multimodal rows are image questions over locally registered project-generated charts, and roughly 600k train rows carry teacher-style soft label distributions.
Data categories:
- service decision arcs — user-intent routing and case triage (refund, logistics, account actions), policy and adequacy judgment;
- language judgment — sentiment, sarcasm, NLI/factuality and paraphrase-style checks;
- knowledge and reasoning QA — multiple-choice knowledge, multi-hop reasoning, arithmetic and math word problems;
- retrieval and selection — document and tabular retrieval, product/route selection;
- multimodal reading — chart/graph and small-graphic question answering;
- calibration-style rows — probability and ordinal confidence judgments.
Training parameters (from the training receipts):
- distributed FSDP on 64 GPUs (4 nodes x 16 GPUs);
- effective batch 256 rows per step, learning rate 2e-06;
- two independent runs: 2,446 steps (sequence cap 8,192) and 700 steps (sequence cap 2,048);
- objective
valid_option_soft_cross_entropyover valid options only (noul_tie_policy = true_on_equal_probability,choice_permutation = global-reference, vision tower trainable); - precision policy fp32-master-bf16; export dtype BF16, readout FP32, temperature 1.0.
Package artifacts
Weight files
| File | Size | SHA-256 |
|---|---|---|
model-00001-of-00002.safetensors |
4,973,336,472 | 27b985a94ccf63aace88b2ff67c1ea0871ce745a49b22b9cda43ea4d32297d8a |
model-00002-of-00002.safetensors |
4,107,897,400 | 81f61ce9daead4be7e3d950cf027f63f16f757a176407f020406190dc2068bc0 |
tokenizer.json |
19,989,492 | 6f32ce20dc35f57a7f9ad1eac03525bd7d30f9df8cea6507e958279cc3657706 |
model.safetensors.index.json |
68,426 | bda09c55c0152d7a6fb2db6898aea70e728dc528df6f9c026cefb0bd5d1d673b |
Evaluation results
Both evaluations ran on 2026-10-15 on self-hosted compute (64 GPUs) through the
model's own HTTP /v1/systemone wire contract. No result here
is an official leaderboard or board submission; the harnesses enforce no-truncation,
no option filtering, one fixed rendering, and coverage-adjusted scoring. Latency is
per-request server wall time on internal accelerators and is not the sub-1-second
latency-admission measurement of any external board.
Evidence directories keep the run names under which they were created during the v1
execution (lemondeeder-eval-20261015/); no shipped artifact in the model packages
carries that legacy spelling.
1) Decision Index, edition 0.3 — FULL public suite (complete run)
⚙️ 140,620 requests issued across 64 shards (frozen suite-0.3 staging, official
scorer, coverage-adjusted, chance-corrected) → 140,178 scored requests,
complete: true.
| Headline | Value |
|---|---|
| Decision Index (public 0.3) | 50.31 |
| Raw index | 62.47 |
| Median / mean / p95 request latency | 50.4 ms / 205.6 ms / 756.2 ms |
The public index averages the native, chance-corrected, coverage-adjusted score of 37 benchmarks in five areas with fixed weights (knowledge 25.8%, language 25.8%, retrieval 20.0%, tools 18.3%, arts 10%); inside an area, ★ gold benchmarks weigh 1.2 and the rest 1.0. Unanswered counts as wrong.
Per-area scores
| Area | Raw | Skill (chance-corrected) | Coverage | Benchmarks |
|---|---|---|---|---|
| Knowledge & Reasoning | 0.4895 | 0.3395 | 0.9998 | 11 |
| Language Understanding | 0.6962 | 0.5751 | 1.0000 | 9 |
| Retrieval & Classification | 0.6661 | 0.5494 | 1.0000 | 6 |
| Tools & Automation | 0.7108 | 0.6741 | 0.9986 | 5 |
| Arts & Human Taste | 0.5487 | 0.3340 | 0.9978 | 6 |
Per-benchmark native scores (★ = counted in the public index)
| # | Benchmark | Metric | Requests | Score | Skill | Cov. |
|---|---|---|---|---|---|---|
| 1 | BFCL ★ | case exact accuracy | 1,694 | 0.9416 | 0.9211 | 1.000 |
| 2 | ToolRet ★ | nDCG@10 | 685 | 0.6540 | 0.5982 | 0.997 |
| 3 | API-Bank ★ | accuracy | 508 | 0.8320 | 0.8254 | 0.996 |
| 4 | BANKING77 ★ | macro-F1 | 3,080 | 0.7418 | 0.7385 | 1.000 |
| 5 | CLINC150+OOS ★ | macro-F1 | 5,500 | 0.8634 | 0.8626 | 1.000 |
| 6 | RouterBench | selected quality | 10,000 | 0.7746 | 0.4715 | 1.000 |
| 9 | Home appliance simulator ★ | case exact accuracy | 88 | 0.4091 | 0.4091 | 1.000 |
| 10 | SGD/SGD-X | macro-F1 | 2,500 | 0.4360 | 0.0616 | 1.000 |
| 11 | ContractNLI ★ | macro-F1 | 123 | 0.7389 | 0.6224 | 1.000 |
| 12 | ANLI ★ | macro-F1 | 3,200 | 0.6729 | 0.5100 | 1.000 |
| 20 | BPoMP ★ | accuracy | 5,000 | 0.8718 | 0.7525 | 1.000 |
| 21 | Humicroedit ★ | accuracy | 2,628 | 0.6111 | 0.2222 | 1.000 |
| 22 | POP909-CL ★ | accuracy | 2,000 | 0.1890 | 0.1701 | 0.987 |
| 23 | cfcolor ★ | accuracy | 5,000 | 0.6088 | 0.1827 | 1.000 |
| 24 | MMLU | accuracy | 14,033 | 0.7914 | 0.7219 | 1.000 |
| 25 | GPQA Diamond ★ | accuracy | 196 | 0.4286 | 0.2381 | 1.000 |
| 26 | ARC-Easy | accuracy | 2,376 | 0.9836 | 0.9781 | 1.000 |
| 27 | ARC-Challenge | accuracy | 1,172 | 0.9548 | 0.9397 | 1.000 |
| 28 | WinoGrande ★ | accuracy | 1,267 | 0.8019 | 0.6038 | 1.000 |
| 29 | HellaSwag ★ | accuracy | 10,042 | 0.8520 | 0.8027 | 1.000 |
| 30 | GSM8K ★ | accuracy | 2,638 | 0.5716 | 0.4866 | 1.000 |
| 31 | ChessBench ★ | accuracy | 5,000 | 0.1384 | 0.0616 | 1.000 |
| 32 | MuSR ★ | accuracy | 752 | 0.5731 | 0.3213 | 1.000 |
| 33 | SATA-Bench ★ | case exact accuracy | 1,650 | 0.1539 | 0.1427 | 1.000 |
| 34 | SimpleBench | accuracy | 10 | 0.1000 | 0.0000 | 1.000 |
| 36 | BRIGHT ★ | nDCG@10 | 220 | 0.4444 | 0.3714 | 1.000 |
| 37 | Amazon ESCI ★ | macro-F1 | 5,000 | 0.4827 | 0.3511 | 1.000 |
| 38 | ACOS ★ | per-review F1 | 1,565 | 0.1970 | 0.1713 | 1.000 |
| 39 | FinEntity ★ | macro-F1 | 979 | 0.9381 | 0.9090 | 1.000 |
| 40 | iSarcasmEval ★ | Sarcasm F1 (track A, EN) | 4,600 | 0.5552 | 0.4278 | 1.000 |
| 41 | VAST ★ | macro-F1 | 3,006 | 0.7251 | 0.5876 | 1.000 |
| 42 | NLI4CT ★ | macro-F1 | 5,500 | 0.7997 | 0.6103 | 1.000 |
| 43 | CRUXEval ★ | accuracy | 570 | 0.6035 | 0.3709 | 1.000 |
| 44 | CLadder ★ | accuracy | 5,000 | 0.6846 | 0.3692 | 1.000 |
| 45 | HLE ★ | accuracy | 501 | 0.1140 | 0.0000 | 0.998 |
| 50 | Habermas Machine ★ | accuracy | 1,676 | 0.4063 | 0.1384 | 1.000 |
| 56 | PhishNChips ★ | phishing-decision accuracy | 2,000 | 0.7315 | 0.4630 | 1.000 |
| 57 | MMLU-Pro ★ | accuracy | 12,032 | 0.5780 | 0.5254 | 1.000 |
| 58 | BBH fixed-option tasks ★ | accuracy | 5,507 | 0.7018 | 0.5678 | 1.000 |
| 59 | RAGTruth ★ | F1 on hallucinated class | 2,700 | 0.7601 | 0.5026 | 1.000 |
| 61 | HoVer ★ | claim-verification accuracy | 4,000 | 0.7225 | 0.4450 | 1.000 |
| 62 | When2Call MCQ ★ | accuracy | 3,652 | 0.6528 | 0.5371 | 1.000 |
| 64 | New Yorker caption matching ★ | accuracy | 528 | 0.6307 | 0.5384 | 1.000 |
Evidence: lemondeeder-eval-20261015/di/runs/20261015-lemondecider-4b-di-full-0.3/
(64 shard dirs + merged/results.jsonl + official scores.json, index.json,
benchmark-summary.json).
2) JevBench public-231 (typed-decision public sanity suite)
231/231 items attempted and scorable, complete, native probability output,
dataset hash-pinned (dc3995d8…de51, jevbench-v1 harness, typesafe adapter).
| Metric | Value |
|---|---|
| Accuracy | 83.12% (192/231) |
| Macro accuracy | 84.55% |
| Brier (mean) | 0.2392 |
| Ordinal MAE (score items) | 0.1207 |
| Paraphrase consistency | 36/36 pairs agree (100%) |
| Coverage / operational success / schema validity | 1.0 / 1.0 / 1.0 (strict 1.0) |
| Latency p50 / p95 | 47.1 ms / 340.3 ms |
Per-family results
| Family | n | Accuracy | Brier |
|---|---|---|---|
| adequacy | 12 | 1.000 | 0.051 |
| adversarial | 6 | 1.000 | 0.002 |
| ambiguous | 7 | 0.571 | 0.661 |
| extraction | 24 | 1.000 | 0.000 |
| fact | 12 | 1.000 | 0.000 |
| intent | 24 | 1.000 | 0.001 |
| judge_hard | 17 | 0.647 | 0.499 |
| long_policy | 19 | 0.579 | 0.585 |
| multi_hop | 18 | 0.722 | 0.481 |
| ordinal | 12 | 1.000 | 0.000 |
| policy | 12 | 1.000 | 0.007 |
| probability | 10 | 0.800 | 0.338 |
| routing | 12 | 1.000 | 0.013 |
| routing_hard | 5 | 1.000 | 0.000 |
| temporal_numeric | 15 | 0.067 | 1.006 |
| tool_selection | 12 | 1.000 | 0.000 |
| tradeoff | 6 | 0.833 | 0.450 |
| trap | 8 | 1.000 | 0.038 |
Known weak spots on this suite: long-policy judgement, ambiguous states, hard-judge calls, and especially temporal/numeric reasoning (6.7%), which the model inherits from its parents' training data.
Evidence: lemondeeder-eval-20261015/jevbench/20261015-lemondecider-4b-jevbench-public231/
(results.jsonl, ledger.jsonl, public-summary.json, 5-item smoke gate passed).
Usage
Dependencies: transformers==5.17.0 (needs the Qwen3.5 hybrid classes),
flash-linear-attention==0.5.2, pillow, torch with SDPA.
This package honours a single runtime env var: LEMONDECIDER_CUDNN_SDPA=1
(keep the cuDNN SDPA backend on CUDA; defaults are kept on CPU/ROCm).
import sys, torch
from transformers import AutoModel
ckpt = "lemondecider-4b-20261015-v2" # local package directory
sys.path.insert(0, ckpt) # make the in-package runtime importable
from decision_utils import load_processor, predict_batch
model = AutoModel.from_pretrained(
ckpt, trust_remote_code=True, local_files_only=True,
dtype=torch.bfloat16, device_map="cuda:0", attn_implementation="sdpa",
).eval()
processor = load_processor(ckpt)
rows = [{
"state": "The user asks how to return a damaged order placed last week.",
"question": {"type": "choice",
"instructions": "Pick the best next action",
"criteria": {"refund": "Process a refund",
"replace": "Send a replacement",
"escalate": "Escalate to a human"}},
}]
answers = predict_batch(model, processor, rows)
print(answers["answers"][0]) # choice, probabilities over the 3 options, confidence
CLI from inside the package: python3 model.py --checkpoint . --device cuda:0 --input rows.json.
Batching: each predict_batch call runs ONE forward pass over all the rows you pass
(inputs are left-padded to the longest prompt); the HTTP serving bridge chunks its
requests at 8 questions per forward pass. Requests over the 262,144-token
max_input_length limit are rejected with the limit named (HTTP 422 in server
wrappers) — nothing is truncated.
Intended use & limitations
- Intended: typed decision endpoints — routing, intent classification, policy/ adequacy checks, structured QA over explicit option lists, pipeline control.
- The model emits option probabilities only; it is not a generative assistant.
- Ratings (
scoreitems) are supported through the same readout; they inherit the parents' weak arithmetic/timing behavior (see the temporal_numeric row above). - Trained on public data sources assembled by the project; no evaluation rows were used for training (corpus admission contracts enforced this).
- Uncalibrated: temperature 1.0, no post-hoc calibration fitted. The 6 low-confidence bins in the JevBench calibration table show mild over-confidence in the 0.6–0.8 confidence range.
License
LICENSE— Apache-2.0 text as preserved from the parent checkpoint lineage (base: Qwen/Qwen3.5-4B, inherit the base model's terms).LICENSE.code— MIT for runtime/server code components adapted from upstream.- No benchmark score is claimed by this package beyond the reproducible evidence files listed above; the Decision Index owner reserves 80% of its Full score for private tests, so the 50.31 public index is not a board submission.
Acknowledgements
Built by the Lemon project. Runtime and server components build on Apache-2.0 and MIT upstream code (see LICENSE files). Base backbone by the Qwen team.