File size: 9,617 Bytes
dba367c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
---
license: apache-2.0
base_model:
  - Qwen/Qwen3.5-4B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-classification
language:
  - en
  - zh
tags:
  - decision-model
  - system-one
  - decision-index
  - lora-merged
---

# KnowLine-4B-Gen5

**The fifth release of PelaAI's 4B System One decision model.**

[Inference guide](INFERENCE.md) · [中文说明](README.zh.md) · weights Apache-2.0 · previous version:
[KnowLine-4B-Gen4](https://huggingface.co/PelaAI/KnowLine-4B-Gen4)

Highlights:

- **Decision Index 0.3 public suite: 65.67** (self-run; Gen4: 64.90). The board's full score comes from the maintainers'
  private tests; see [Comparison](#comparison).
- **Calibrated probabilities:** a temperature is folded into the weights. Computed with the board's method, ECE is about
  0.04-0.05 and the Brier score about 0.28-0.31; see [Calibration](#calibration).
- **Fast:** one request at a time on one H20, 31.6 ms median (see [INFERENCE.md](INFERENCE.md)).
- **Images without any image training:** the vision tower is the base model's and image input is zero-shot.

KnowLine is an independent model. It is not affiliated with, endorsed by, or derived from TypeSafe or Jev.

## What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model
answers every question with a probability distribution over its options:

- one forward pass per question, with no text generation;
- the output is the probability of each option's label token;
- existing Jev clients only need to change the base URL, and users of earlier versions only need to change the model
  name, because the interface and serving settings are unchanged.

| | |
|---|---|
| Base model | [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) (Apache-2.0) |
| Training | LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights (see [How Gen5 was made](#how-gen5-was-made)) |
| Release format | bf16 weights, with a readout temperature T = 1.50 folded into the final norm weight (see [Calibration](#calibration)). The vision tower and MTP head are unchanged from the base model; the config, tokenizer and chat template are identical to those of Gen1-Gen4. |
| Languages | English, Simplified Chinese, Traditional Chinese |

## Quickstart

```bash
pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-4B-Gen5 0 8080     # SGLang (FP8 at load) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
```

- **Front end:** `knowline_server.py` is a single file that depends only on transformers and requests. It is the same
  file as in Gen4.
- **Full settings:** the exact settings used for our Decision Index run are in [INFERENCE.md](INFERENCE.md).

## How Gen5 was made

- **Base model:** [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B).
- **Training:** one epoch of LoRA SFT (rank 32, alpha 64, language model only), merged into bf16 weights. Training
  examples include decision tasks written and labelled by LLM teachers.
- **Readout temperature:** T = 1.50, fitted on held-out data and folded into the final norm weight. It changes the size
  of the probabilities, but not which option ranks first.

The training data is not included in this repository.

## Comparison

Scores on the Decision Index. Since Decision Index 0.4 (2026-10-19), the board's full score is 60% private tests of the
same skills and 40% private tasks from new domains; the public suite is shown but no longer counts. Our full score is not
available yet.

| model | size | DI 0.3 public | DI 0.4 full | source |
|---|---|---|---|---|
| **KnowLine-4B-Gen5** | 4B | **65.67** | not yet scored | self-run, official kit |
| KnowLine-4B-Gen4 | 4B | 64.90 | not yet scored | self-run, official kit |
| StartLux-Decision v1.1 4B | 4.7B | 67.81 | 56.35 | board |
| Perplexity Decider Flash v1.2 | 4.5B | 60.06 | 55.39 | board |
| vLLM-SR d3-mini | 4.5B | 54.90 | 51.25 | board |
| OneJev 1.6 | 4.5B | 61.95 | 49.54 | board |
| Jev 1.14 | (API) | 60.70 | 62.22 | board |
| KnowLine-35B-A3B-Gen1 | 35B-A3B | 69.49 | 62.28 | board |

Board rows as of the board's latest update (Decision Index 0.4).

Public-suite scores by area (DI 0.3):

| area | KnowLine-4B-Gen5 | KnowLine-4B-Gen4 |
|---|---|---|
| Knowledge & reasoning | 45.0 | 44.6 |
| Language | 66.8 | 65.5 |
| Retrieval & routing | 73.3 | 72.5 |
| Tools & agents | 87.4 | 87.6 |
| Arts & taste | 61.2 | 59.2 |

## Releases

The model is trained in a self-evolving loop, and new versions will follow.

| model | date | DI 0.3 public | notes |
|---|---|---|---|
| KnowLine-4B-Gen5 (this model) | 2026-10-21 | 65.67 | fifth release |
| [KnowLine-4B-Gen4](https://huggingface.co/PelaAI/KnowLine-4B-Gen4) | 2026-10-10 | 64.90 | fourth release |
| [KnowLine-4B-Gen3](https://huggingface.co/PelaAI/KnowLine-4B-Gen3) | 2026-10-09 | 63.11 | third release |
| [KnowLine-4B-Gen2](https://huggingface.co/PelaAI/KnowLine-4B-Gen2) | 2026-10-08 | 62.54 | second release |
| [KnowLine-4B-Gen1](https://huggingface.co/PelaAI/KnowLine-4B-Gen1) | 2026-10-07 | 60.47 | first release |

## Evaluation

All results are self-run and have not been verified by a third party.

### Other public benchmarks

| suite | Gen5 | Gen4 |
|---|---|---|
| C-Eval (4 categories, macro) | 79.2 | 78.7 |
| JevBench / JevBench-hard | 83.1 / 66.7 | 86.1 / 72.1 |

### Web operation (Mind2Web official test splits, evaluation only)

| split | Gen5 element selection | Gen5 operation (balanced) | Gen4 element selection | Gen4 operation (balanced) |
|---|---|---|---|---|
| test_task (Cross-Task) | 92.6 | 97.1 | 92.9 | 97.0 |
| test_website (Cross-Website) | 90.9 | 96.9 | 91.9 | 95.7 |
| test_domain (Cross-Domain) | 91.8 | 98.4 | 91.7 | 97.2 |

- About the same as Gen4; the differences are within noise.
- The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the
  page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web
  leaderboard.

### Images (zero-shot)

- **No image training:** Gen5 keeps the base model's vision tower unchanged and saw no images in training.
  `knowline_server.py` accepts OpenAI-style `image` / `image_url` content parts in a chat state and passes them to
  SGLang; each question is still one prefill.

### Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or
wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are
used.

| metric | Gen5 | Gen4 |
|---|---|---|
| ECE (lower is better) | 0.04-0.05 | 0.04-0.05 |
| Brier score (lower is better) | 0.28-0.31 | 0.29-0.31 |
| Confidence ≥95% but wrong | 1.5-1.6% | 1.6-1.8% |
| Mean confidence / accuracy | 0.82-0.83 / 0.78-0.80 | 0.82 / 0.77 |

- **Temperature in the weights:** the final norm weight includes a readout temperature T = 1.50, so every logit is
  divided by 1.50 before the softmax over the option labels. The top-ranked option does not change; the probabilities
  are just less extreme. T was fitted on held-out data.
- **Why ranges:** the board does not publish which 32 benchmarks it uses, so we report ranges over the plausible
  benchmark sets, as for the earlier releases.
- **Recommendation:** if your use depends on probability thresholds, check calibration on your own data.

## Disclosures

- **Decontamination:** all training data was checked against every Decision Index 0.3 row and the public benchmarks we
  report (Intern-Decision, S1MB, JevBench, MMDM). About 22,000 rows were removed: rows that nearly
  duplicate a benchmark item, repeat its question or claim, or contain its whole state, and, in a stricter second pass,
  most rows that share a source passage with a benchmark item.
- **Teacher-labelled synthetic data:** LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but
  not all of them were checked by a human.
- **Public suite:** the Decision Index public suite is public; the board's private tests may show a lower score.

## Limitations

- **Knowledge-heavy reasoning:** weaker than larger models. The knowledge area score is 45.0 (DI 0.3); a 4B model is
  limited here, and the new-domain part of the board favours larger models.
- **JevBench-hard:** 66.7 against 72.1 for Gen4; the drop is not significant, but check it if your use is close to
  JevBench-hard.
- **New domains:** the board's private new-domain tasks favour larger models; check the model on your own domain.
- **Prompt injection:** an instruction planted in the state can change the answer. Keep untrusted text clearly
  delimited.
- **Images:** there was no image training; image use is zero-shot, so check it on your own data before relying on it.

## Citation

```bibtex
@misc{knowline4bgen5,
  title  = {KnowLine-4B-Gen5: a 4B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-4B-Gen5}
}
```

## License

- **Weights:** Apache-2.0, the same as the base model.
- **Code:** `knowline_server.py` is MIT (see the file header).