KnowLine-31B-Gen1.5: KnowLine-31B-Gen1 weights with the question header for long requests in the front end

8e56aa1 verified 1 day ago

17.1 kB

# KnowLine-31B-Gen1.5: serving and Decision Index reproduction

Gen1.5 is KnowLine-31B-Gen1 with a new way of serving long requests: the weights are the same files, requests that fit
the read-twice limit are read twice as in Gen1, and longer text-only requests get a question header (see
[Question header for long requests](#question-header-for-long-requests)). These settings reproduce our self-run Decision
Index 0.3 evaluation of Gen1.5, scored 2026-10-28 19:22 CST. That run was not sent fresh in full: the requests whose
prompts are the same under the Gen1 and Gen1.5 front ends were carried over from our KnowLine-31B-Gen1 run, and the
requests that change were sent fresh through the Gen1.5 release front end; see [Decision Index run](#decision-index-run).

## Weights

- The weights are byte-identical to [KnowLine-31B-Gen1](https://huggingface.co/PelaAI/KnowLine-31B-Gen1): every weight,
  config and tokenizer file has the same SHA-256 as there (`SHA256SUMS`). They were made in three steps (bf16, about
  63 GB):
  1. **Fine-tuning:** gemma-4-31B-it trained for one epoch with a LoRA adapter (rank 32, alpha 64, language model
     only), merged into the weights.
  2. **Averaging with the base model:** every tensor is 0.75 × the fine-tuned tensor + 0.25 × the gemma-4-31B-it tensor
     (computed in fp32, stored in bf16). Tensors that fine-tuning did not change, such as the vision tower, stay those
     of the base model.
  3. **Readout temperature T = 1.44**, folded in two places so that every final logit is divided exactly by T:
     - the final norm weight `model.language_model.norm.weight` is divided by 1.44 (Gemma 4's RMSNorm applies the
       weight as `x * w`, not `(1 + w) * x`);
     - `text_config.final_logit_softcapping` in `config.json` is 20.833333333333336 instead of 30.0. Gemma 4 caps its logits as
       `cap * tanh(z / cap)`; with both changes the output is `30 * tanh(z / 30) / T`, the capped logit divided by T.
       **Do not change the soft-cap back to 30**, or the temperature is only partly applied.
     - T was fitted by negative log-likelihood on our in-training evaluation sets, served read twice through the Gen1
       front end (4,965 questions); refitting on the released weights gives T = 1.00, and ECE on our held-out sets
       (6,995 questions) is 0.044 (0.102 at T = 1 before folding).
- The architecture is `Gemma4ForConditionalGeneration`. Apart from `final_logit_softcapping`, `config.json` is identical
  to that of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it); the tokenizer files, the chat
  template, `processor_config.json` and `generation_config.json` are byte-identical to the base model.
- `SHA256SUMS` lists every file. Only the serving files (`knowline_server.py`, `knowline_engine.py`, `serve_knowline.sh`)
  and the documentation differ from Gen1; `knowline_layout.py` and `temperatures.json` are the same.
## Software

| component | version |
|---|---|
| SGLang | 0.5.21 (torch 2.13.0, CUDA 13.0) |
| transformers | 5.12.1 |
| front end | `knowline_server.py` + `knowline_layout.py` (this repository; need only transformers and requests) |
| decision-index kit | 0.3 at commit 62d2f51de34a2de64906345b6bc3e98e27ff55c7 |

### About the front end

`knowline_server.py` is the `/v1/systemone` front end of our runs; the prompt layout lives in `knowline_layout.py` next to
it. It is the Gen1 front end with one addition, the question header for long requests (`--header-when-long`).
Requests that read twice covers, and requests with images, are rendered exactly as by the Gen1 front end, prompt for
prompt. It renders with the model's own chat template. The front end contains:

- rendering: the chat template with thinking off; the state is followed by one user turn with the instruction, the
  question and all options labelled A, B, ...; the assistant turn opens with `Answer:`;
- read twice (on by default, see below);
- the question header for text-only requests that read twice does not cover (on in `serve_knowline.sh`, see below);
- label-token scoring: one prefill per question, then a softmax over the option labels only;
- the HTTP server.

### Read twice

- **What it does:** for a text-only request of at most 3,000 tokens (counted with the model's tokenizer over the whole
  request read once), every turn after the system turn is written a second time before the answer is read: the state, the
  instruction, the question and its options. The second copy starts with the line "Read the same input again before
  answering:". The answer is still read from the label token after `Answer:`, one prefill per question.
- **What it leaves alone:** requests over 3,000 tokens are not repeated: they get the question header (next section) or,
  past its limit, are sent once as in Gen1. Requests with images are sent once. In our Decision Index run, 132,376 of the
  140,620 requests (94.1%) were read twice, the same requests as in the Gen1 run.
- **Switches:** `--no-read-twice` reads every request once, without the question header; `--twice-max-tokens` sets the
  cut. `knowline_engine.py` takes `--option read_twice=0` and `--option twice_max_tokens=<n>` for the same. All our
  evaluations of this model used read twice.
- **Cost:** about 1.54 times the prompt tokens of reading once for the requests it covers (over the whole suite, 323
  million against 209 million with the Gen1 serving, the same rows and tokenizer as KnowLine-26B-A4B-Gen2.5's run); see
  [Hardware and latency](#hardware-and-latency).
- **Credit:** the read-twice layout follows FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT),
  including its "Read the same input again before answering:" line; repeating the input goes back to RE2 (Xu et al.,
  2023, [arXiv:2309.06275](https://arxiv.org/abs/2309.06275)). `knowline_layout.py` is our own implementation.

### Question header for long requests

- **What it does:** for a text-only request that read twice does not cover (more than 3,000 tokens read once), a header
  goes before the state: the line "Questions that will be asked about the conversation or state below:", then one line
  "- <question>" per question of the request, in request order and without the options. The request is then read once;
  the instruction, the question block, the options and `Answer:` are as in Gen1, and the answer is read from the label
  token, one prefill per question. Requests with a single question get the header too.
- **Limit:** the header is used only if the longest of the request's prompts with it is at most 32,768 tokens (counted
  with the model's tokenizer); otherwise the request keeps the Gen1 read-once prompts.
- **What it leaves alone:** requests that read twice covers and requests with images are byte-identical to Gen1. In our
  Decision Index run, 7,845 of the 140,620 requests (5.6%) got the header, and 399 (0.3%) were over the limit and were
  sent as in Gen1.
- **Switches:** `serve_knowline.sh` passes `--header-when-long 32768`. `knowline_server.py` started on its own defaults to
  `--header-when-long 0`, which gives exactly the Gen1 prompts. `knowline_engine.py` uses 32768 by default;
  `--option header_when_long=0` turns it off.
- **Cost:** the header is part of every question's prompt, and requests with many questions have long headers. Over the
  whole suite the front end's prompt tokens, summed over questions, are 896 million, against 323 million with the Gen1
  serving and 209 million reading every request once. With SGLang's prefix cache the header and the state are prefilled
  once per request; counted that way over all suite requests, the prefilled tokens are about 5% more than with the Gen1
  serving.
- **Credit:** the question listing follows FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT),
  with "conversation or state" in place of its "context". Aleria Judge ([github.com/ALERIA-ML/aleria-judge](https://github.com/ALERIA-ML/aleria-judge), Apache-2.0) uses the same header structure.

The front end uses temperature 1 and the readout temperature is in the weights, so both backends get it. As in Gen1,
`temperatures.json` (`{"score": 0.6944}`, shipped in this repository) makes the front end divide the logits of score
(rubric) questions by a further 0.6944, so they are read at an effective temperature of 1.44 × 0.6944 = 1.0, while
yes/no and choice questions stay at 1.44. Score questions are often read as an expected value over the scale, which a
higher temperature pulls toward the middle. `serve_knowline.sh` passes the file and `knowline_engine.py` loads it by
default (`--option temperatures=none` turns it off). KnowLine releases before Gen1 did not use per-type temperatures.
All our evaluations of this model used them, except where noted.

## Serve

`serve_knowline.sh` starts both processes, with the settings of KnowLine-31B-Gen1 plus `--header-when-long 32768`.

1. **SGLang engine, bf16.**
   - Served in bf16, without `--quantization fp8`, as the 26B-A4B (for which two FP8 runs of the same questions agreed
     on only 95.7% of the decisions, against 99.35% in bf16). We did not test FP8 at load for this model.
   - There is no `--mamba-radix-cache-strategy` flag (Gemma 4 has no recurrent layers).
   - SGLang reserves `--mem-fraction-static` (default 0.85) of the GPU's memory for the weights and its KV cache: about
     82 GB on a 96 GB card, of which the weights take about 63 GB. The value only sizes the KV cache; scores do not
     depend on it. Every run below used one 96 GB H20 per server; we have not run it on smaller GPUs.
   - SGLang treats the model as multimodal by itself. The Decision Index run sent text only.
2. **`/v1/systemone` front end:** `knowline_server.py`, `chat` style, temperature 1, 16 scoring threads, read twice up
   to 3,000 tokens, the question header for longer text-only requests whose prompts fit 32,768 tokens
   (`--header-when-long 32768`), per-type temperatures from `temperatures.json` (score questions / 0.6944), box
   questions with `--box-prompt gemma --box-coords yxyx` (Gemma's `box_2d` format, y first) and `box_calibration.json`
   when present.
```bash
pip install "sglang==0.5.21" "transformers==5.12.1" requests   # torch / CUDA per SGLang's install docs
bash serve_knowline.sh PelaAI/KnowLine-31B-Gen1.5 0 8080          # GPU 0; SGLang on :9080, /v1/systemone on :8080
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
```

Equivalent manual launch, with the exact flags of our run:

```bash
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path PelaAI/KnowLine-31B-Gen1.5 --served-model-name m \
  --tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head --port 9080 &
python knowline_server.py --model PelaAI/KnowLine-31B-Gen1.5 --backend sglang --url http://127.0.0.1:9080 \
  --served-model-name m --temperature 1 --workers 16 --port 8080 --temperatures temperatures.json \
  --header-when-long 32768 --box-prompt gemma --box-coords yxyx          # read twice is on by default
```

Without SGLang (through transformers; text only, no prefix cache, bf16):

```bash
pip install torch "transformers==5.12.1" requests
python knowline_server.py --model PelaAI/KnowLine-31B-Gen1.5 --backend hf --port 8080 --temperatures temperatures.json \
  --header-when-long 32768
```

- Loaded in bf16, the weights take about 62.5 GB (58.3 GiB) of GPU memory.
- Checked on one H20 with the Gen1 front end (read twice on, no question header); we did not repeat this check for
  Gen1.5: on 30 random Decision Index 0.3 rows, 80 of the 82 answers were the same as in our SGLang run; the other two
  are near-ties (the top two options within 0.06 of each other in one of the two runs). Requests took 576.5 ms median one
  at a time.
- Images need the SGLang backend.

## Decision Index run

- Suite rows: the 140,620 rows that `decision-index run --edition 0.3` sends: `selected-rows.jsonl.gz`,
  `added-rows.jsonl.gz` and `gsm8k-rows.jsonl.gz` of `suite-0.3`, with ToolRet, BRIGHT and ACOS cut to their scoring
  subsets.
- **Carried over (132,376 rows):** every request that read twice covers has the same prompts under the Gen1 and Gen1.5
  front ends (the new code path runs only for requests that read twice does not cover; a byte-for-byte check of the
  rendered prompts on a random sample of requests of every kind found no difference). Their results were taken from our
  KnowLine-31B-Gen1 run (same weights, same SGLang flags, scored 2026-10-24 04:28 CST), unchanged.
- **Sent fresh (8,244 rows):** every request that read twice does not cover, sent through this repository's
  `serve_knowline.sh` on four identical servers, one per H20 (`--mem-fraction-static 0.90`, which only sizes the KV
  cache, as in the Gen1 run). Each part is a resumable `decision-index run` against its server. None failed and nothing
  was retried.
- The two groups were concatenated and scored with the kit: all 140,178 scoreable requests were answered; none was
  unsupported or truncated, and none failed.
- **Consistency check:** 830 rows (2,808 questions), sampled by benchmark from both groups, were sent again through the
  full Gen1.5 front end: 2,801 of the 2,808 questions (99.75%) got the same top answer as in the scored results (carried
  rows 1,283 of 1,289, fresh rows 1,518 of 1,519).

```bash
decision-index run --edition 0.3 --engine http --option base_url=http://127.0.0.1:<port> --option model=m --no-verify \
  --compact --rows parts/<i>.jsonl.gz --out parts/<i>          # the 8,244 rows, run in parallel
cat <carried results.jsonl> parts/*/results.jsonl > results.jsonl
decision-index score --suite-dir suite-0.3 --results results.jsonl --engine http --out score
```

## Hardware and latency

- Four NVIDIA H20 (96 GB each), one server per GPU.
- **Latency:** we sent 700 Decision Index 0.3 rows, sampled in the style of the board's latency-v1 method (per-catalog
  counts, a systematic sample by payload length), one at a time to one server on one H20 with the release settings
  (after 10 warm-up rows).
  - Requests took 330.3 ms median, 1,135.1 ms mean, 1,417.7 ms at the 80th percentile and 4,240.9 ms at the 95th.
  - 1,602 of the 1,605 answers were the same as in our Decision Index results.
  - The Gen1 card's latency (340.3 ms median, 910.0 ms mean) and KnowLine-26B-A4B-Gen2.5's (66.3 ms median, 151.1 ms
    mean) were measured on a different sample of 300 rows, so they are not directly comparable.
- Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1).
## One-command Decision Index run

`knowline_engine.py` (in this repository) is a Decision Index engine. It runs the same front end in process, so there
is no server to start by hand:

```bash
pip install "sglang==0.5.21" "transformers==5.12.1" requests        # plus the decision-index kit
git clone https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 && cd KnowLine-31B-Gen1.5   # puts the three files on the path
python -m decision_index pipeline --engine knowline_engine:KnowLine \
    --option model=PelaAI/KnowLine-31B-Gen1.5 --option revision=<commit> --out runs/KnowLine-31B-Gen1.5
```

- **Default backend and layout (the setting of our runs):** read twice is on and the question header is on for longer
  requests (`header_when_long=32768`); the engine starts SGLang with the flags above (bf16,
  `--mem-fraction-static 0.85`) on a free local port and stops it when the run ends. Use `--option gpu=<n>` to pick a
  GPU, and `--option sglang_python=<python>` if SGLang lives in another environment. `--option header_when_long=0` gives
  the Gen1 prompts.
- **`--option backend=hf`:** transformers only, bf16, no server; slower, and not the setting of our runs.
- **Limits:** up to 64 questions per request and 255 options per question. Larger requests are reported as
  unsupported; nothing is truncated.
- **Temperature in both backends:** transformers and SGLang both read the final norm weight and
  `final_logit_softcapping` from these files, so both backends apply the folded temperature; the per-type temperatures
  of `temperatures.json` are applied by the front end, the same for both. Our Decision Index run and
  calibration numbers use SGLang.

## Acknowledgements

- Read-twice layout and question header: FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT
  licence).
- Question header: also used by Aleria Judge ([github.com/ALERIA-ML/aleria-judge](https://github.com/ALERIA-ML/aleria-judge), Apache-2.0).
- Input repetition: Xu et al., 2023, "Re-Reading Improves Reasoning in Large Language Models"
  ([arXiv:2309.06275](https://arxiv.org/abs/2309.06275)).

	# KnowLine-31B-Gen1.5: serving and Decision Index reproduction

	Gen1.5 is KnowLine-31B-Gen1 with a new way of serving long requests: the weights are the same files, requests that fit
	the read-twice limit are read twice as in Gen1, and longer text-only requests get a question header (see
	[Question header for long requests](#question-header-for-long-requests)). These settings reproduce our self-run Decision
	Index 0.3 evaluation of Gen1.5, scored 2026-10-28 19:22 CST. That run was not sent fresh in full: the requests whose
	prompts are the same under the Gen1 and Gen1.5 front ends were carried over from our KnowLine-31B-Gen1 run, and the
	requests that change were sent fresh through the Gen1.5 release front end; see [Decision Index run](#decision-index-run).

	## Weights

	- The weights are byte-identical to [KnowLine-31B-Gen1](https://huggingface.co/PelaAI/KnowLine-31B-Gen1): every weight,
	config and tokenizer file has the same SHA-256 as there (`SHA256SUMS`). They were made in three steps (bf16, about
	63 GB):
	1. Fine-tuning: gemma-4-31B-it trained for one epoch with a LoRA adapter (rank 32, alpha 64, language model
	only), merged into the weights.
	2. Averaging with the base model: every tensor is 0.75 × the fine-tuned tensor + 0.25 × the gemma-4-31B-it tensor
	(computed in fp32, stored in bf16). Tensors that fine-tuning did not change, such as the vision tower, stay those
	of the base model.
	3. Readout temperature T = 1.44, folded in two places so that every final logit is divided exactly by T:
	- the final norm weight `model.language_model.norm.weight` is divided by 1.44 (Gemma 4's RMSNorm applies the
	weight as `x * w`, not `(1 + w) * x`);
	- `text_config.final_logit_softcapping` in `config.json` is 20.833333333333336 instead of 30.0. Gemma 4 caps its logits as
	`cap * tanh(z / cap)`; with both changes the output is `30 * tanh(z / 30) / T`, the capped logit divided by T.
	Do not change the soft-cap back to 30, or the temperature is only partly applied.
	- T was fitted by negative log-likelihood on our in-training evaluation sets, served read twice through the Gen1
	front end (4,965 questions); refitting on the released weights gives T = 1.00, and ECE on our held-out sets
	(6,995 questions) is 0.044 (0.102 at T = 1 before folding).
	- The architecture is `Gemma4ForConditionalGeneration`. Apart from `final_logit_softcapping`, `config.json` is identical
	to that of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it); the tokenizer files, the chat
	template, `processor_config.json` and `generation_config.json` are byte-identical to the base model.
	- `SHA256SUMS` lists every file. Only the serving files (`knowline_server.py`, `knowline_engine.py`, `serve_knowline.sh`)
	and the documentation differ from Gen1; `knowline_layout.py` and `temperatures.json` are the same.

	## Software

	\| component \| version \|
	\|---\|---\|
	\| SGLang \| 0.5.21 (torch 2.13.0, CUDA 13.0) \|
	\| transformers \| 5.12.1 \|
	\| front end \| `knowline_server.py` + `knowline_layout.py` (this repository; need only transformers and requests) \|
	\| decision-index kit \| 0.3 at commit 62d2f51de34a2de64906345b6bc3e98e27ff55c7 \|

	### About the front end

	`knowline_server.py` is the `/v1/systemone` front end of our runs; the prompt layout lives in `knowline_layout.py` next to
	it. It is the Gen1 front end with one addition, the question header for long requests (`--header-when-long`).
	Requests that read twice covers, and requests with images, are rendered exactly as by the Gen1 front end, prompt for
	prompt. It renders with the model's own chat template. The front end contains:

	- rendering: the chat template with thinking off; the state is followed by one user turn with the instruction, the
	question and all options labelled A, B, ...; the assistant turn opens with `Answer:`;
	- read twice (on by default, see below);
	- the question header for text-only requests that read twice does not cover (on in `serve_knowline.sh`, see below);
	- label-token scoring: one prefill per question, then a softmax over the option labels only;
	- the HTTP server.

	### Read twice

	- What it does: for a text-only request of at most 3,000 tokens (counted with the model's tokenizer over the whole
	request read once), every turn after the system turn is written a second time before the answer is read: the state, the
	instruction, the question and its options. The second copy starts with the line "Read the same input again before
	answering:". The answer is still read from the label token after `Answer:`, one prefill per question.
	- What it leaves alone: requests over 3,000 tokens are not repeated: they get the question header (next section) or,
	past its limit, are sent once as in Gen1. Requests with images are sent once. In our Decision Index run, 132,376 of the
	140,620 requests (94.1%) were read twice, the same requests as in the Gen1 run.
	- Switches: `--no-read-twice` reads every request once, without the question header; `--twice-max-tokens` sets the
	cut. `knowline_engine.py` takes `--option read_twice=0` and `--option twice_max_tokens=<n>` for the same. All our
	evaluations of this model used read twice.
	- Cost: about 1.54 times the prompt tokens of reading once for the requests it covers (over the whole suite, 323
	million against 209 million with the Gen1 serving, the same rows and tokenizer as KnowLine-26B-A4B-Gen2.5's run); see
	[Hardware and latency](#hardware-and-latency).
	- Credit: the read-twice layout follows FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT),
	including its "Read the same input again before answering:" line; repeating the input goes back to RE2 (Xu et al.,
	2023, [arXiv:2309.06275](https://arxiv.org/abs/2309.06275)). `knowline_layout.py` is our own implementation.

	### Question header for long requests

	- What it does: for a text-only request that read twice does not cover (more than 3,000 tokens read once), a header
	goes before the state: the line "Questions that will be asked about the conversation or state below:", then one line
	"- <question>" per question of the request, in request order and without the options. The request is then read once;
	the instruction, the question block, the options and `Answer:` are as in Gen1, and the answer is read from the label
	token, one prefill per question. Requests with a single question get the header too.
	- Limit: the header is used only if the longest of the request's prompts with it is at most 32,768 tokens (counted
	with the model's tokenizer); otherwise the request keeps the Gen1 read-once prompts.
	- What it leaves alone: requests that read twice covers and requests with images are byte-identical to Gen1. In our
	Decision Index run, 7,845 of the 140,620 requests (5.6%) got the header, and 399 (0.3%) were over the limit and were
	sent as in Gen1.
	- Switches: `serve_knowline.sh` passes `--header-when-long 32768`. `knowline_server.py` started on its own defaults to
	`--header-when-long 0`, which gives exactly the Gen1 prompts. `knowline_engine.py` uses 32768 by default;
	`--option header_when_long=0` turns it off.
	- Cost: the header is part of every question's prompt, and requests with many questions have long headers. Over the
	whole suite the front end's prompt tokens, summed over questions, are 896 million, against 323 million with the Gen1
	serving and 209 million reading every request once. With SGLang's prefix cache the header and the state are prefilled
	once per request; counted that way over all suite requests, the prefilled tokens are about 5% more than with the Gen1
	serving.
	- Credit: the question listing follows FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT),
	with "conversation or state" in place of its "context". Aleria Judge ([github.com/ALERIA-ML/aleria-judge](https://github.com/ALERIA-ML/aleria-judge), Apache-2.0) uses the same header structure.

	The front end uses temperature 1 and the readout temperature is in the weights, so both backends get it. As in Gen1,
	`temperatures.json` (`{"score": 0.6944}`, shipped in this repository) makes the front end divide the logits of score
	(rubric) questions by a further 0.6944, so they are read at an effective temperature of 1.44 × 0.6944 = 1.0, while
	yes/no and choice questions stay at 1.44. Score questions are often read as an expected value over the scale, which a
	higher temperature pulls toward the middle. `serve_knowline.sh` passes the file and `knowline_engine.py` loads it by
	default (`--option temperatures=none` turns it off). KnowLine releases before Gen1 did not use per-type temperatures.
	All our evaluations of this model used them, except where noted.

	## Serve

	`serve_knowline.sh` starts both processes, with the settings of KnowLine-31B-Gen1 plus `--header-when-long 32768`.

	1. SGLang engine, bf16.
	- Served in bf16, without `--quantization fp8`, as the 26B-A4B (for which two FP8 runs of the same questions agreed
	on only 95.7% of the decisions, against 99.35% in bf16). We did not test FP8 at load for this model.
	- There is no `--mamba-radix-cache-strategy` flag (Gemma 4 has no recurrent layers).
	- SGLang reserves `--mem-fraction-static` (default 0.85) of the GPU's memory for the weights and its KV cache: about
	82 GB on a 96 GB card, of which the weights take about 63 GB. The value only sizes the KV cache; scores do not
	depend on it. Every run below used one 96 GB H20 per server; we have not run it on smaller GPUs.
	- SGLang treats the model as multimodal by itself. The Decision Index run sent text only.
	2. `/v1/systemone` front end: `knowline_server.py`, `chat` style, temperature 1, 16 scoring threads, read twice up
	to 3,000 tokens, the question header for longer text-only requests whose prompts fit 32,768 tokens
	(`--header-when-long 32768`), per-type temperatures from `temperatures.json` (score questions / 0.6944), box
	questions with `--box-prompt gemma --box-coords yxyx` (Gemma's `box_2d` format, y first) and `box_calibration.json`
	when present.

	```bash
	pip install "sglang==0.5.21" "transformers==5.12.1" requests # torch / CUDA per SGLang's install docs
	bash serve_knowline.sh PelaAI/KnowLine-31B-Gen1.5 0 8080 # GPU 0; SGLang on :9080, /v1/systemone on :8080
	curl -s http://127.0.0.1:8080/health
	curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
	"model": "m",
	"state": "Customer: my order arrived broken, I want my money back.",
	"questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
	"tone": {"type": "choice", "instructions": "Customer tone?",
	"criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
	```

	Equivalent manual launch, with the exact flags of our run:

	```bash
	CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path PelaAI/KnowLine-31B-Gen1.5 --served-model-name m \
	--tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head --port 9080 &
	python knowline_server.py --model PelaAI/KnowLine-31B-Gen1.5 --backend sglang --url http://127.0.0.1:9080 \
	--served-model-name m --temperature 1 --workers 16 --port 8080 --temperatures temperatures.json \
	--header-when-long 32768 --box-prompt gemma --box-coords yxyx # read twice is on by default
	```

	Without SGLang (through transformers; text only, no prefix cache, bf16):

	```bash
	pip install torch "transformers==5.12.1" requests
	python knowline_server.py --model PelaAI/KnowLine-31B-Gen1.5 --backend hf --port 8080 --temperatures temperatures.json \
	--header-when-long 32768
	```

	- Loaded in bf16, the weights take about 62.5 GB (58.3 GiB) of GPU memory.
	- Checked on one H20 with the Gen1 front end (read twice on, no question header); we did not repeat this check for
	Gen1.5: on 30 random Decision Index 0.3 rows, 80 of the 82 answers were the same as in our SGLang run; the other two
	are near-ties (the top two options within 0.06 of each other in one of the two runs). Requests took 576.5 ms median one
	at a time.
	- Images need the SGLang backend.

	## Decision Index run

	- Suite rows: the 140,620 rows that `decision-index run --edition 0.3` sends: `selected-rows.jsonl.gz`,
	`added-rows.jsonl.gz` and `gsm8k-rows.jsonl.gz` of `suite-0.3`, with ToolRet, BRIGHT and ACOS cut to their scoring
	subsets.
	- Carried over (132,376 rows): every request that read twice covers has the same prompts under the Gen1 and Gen1.5
	front ends (the new code path runs only for requests that read twice does not cover; a byte-for-byte check of the
	rendered prompts on a random sample of requests of every kind found no difference). Their results were taken from our
	KnowLine-31B-Gen1 run (same weights, same SGLang flags, scored 2026-10-24 04:28 CST), unchanged.
	- Sent fresh (8,244 rows): every request that read twice does not cover, sent through this repository's
	`serve_knowline.sh` on four identical servers, one per H20 (`--mem-fraction-static 0.90`, which only sizes the KV
	cache, as in the Gen1 run). Each part is a resumable `decision-index run` against its server. None failed and nothing
	was retried.
	- The two groups were concatenated and scored with the kit: all 140,178 scoreable requests were answered; none was
	unsupported or truncated, and none failed.
	- Consistency check: 830 rows (2,808 questions), sampled by benchmark from both groups, were sent again through the
	full Gen1.5 front end: 2,801 of the 2,808 questions (99.75%) got the same top answer as in the scored results (carried
	rows 1,283 of 1,289, fresh rows 1,518 of 1,519).

	```bash
	decision-index run --edition 0.3 --engine http --option base_url=http://127.0.0.1:<port> --option model=m --no-verify \
	--compact --rows parts/<i>.jsonl.gz --out parts/<i> # the 8,244 rows, run in parallel
	cat <carried results.jsonl> parts/*/results.jsonl > results.jsonl
	decision-index score --suite-dir suite-0.3 --results results.jsonl --engine http --out score
	```

	## Hardware and latency

	- Four NVIDIA H20 (96 GB each), one server per GPU.
	- Latency: we sent 700 Decision Index 0.3 rows, sampled in the style of the board's latency-v1 method (per-catalog
	counts, a systematic sample by payload length), one at a time to one server on one H20 with the release settings
	(after 10 warm-up rows).
	- Requests took 330.3 ms median, 1,135.1 ms mean, 1,417.7 ms at the 80th percentile and 4,240.9 ms at the 95th.
	- 1,602 of the 1,605 answers were the same as in our Decision Index results.
	- The Gen1 card's latency (340.3 ms median, 910.0 ms mean) and KnowLine-26B-A4B-Gen2.5's (66.3 ms median, 151.1 ms
	mean) were measured on a different sample of 300 rows, so they are not directly comparable.
	- Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1).

	## One-command Decision Index run

	`knowline_engine.py` (in this repository) is a Decision Index engine. It runs the same front end in process, so there
	is no server to start by hand:

	```bash
	pip install "sglang==0.5.21" "transformers==5.12.1" requests # plus the decision-index kit
	git clone https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 && cd KnowLine-31B-Gen1.5 # puts the three files on the path
	python -m decision_index pipeline --engine knowline_engine:KnowLine \
	--option model=PelaAI/KnowLine-31B-Gen1.5 --option revision=<commit> --out runs/KnowLine-31B-Gen1.5
	```

	- Default backend and layout (the setting of our runs): read twice is on and the question header is on for longer
	requests (`header_when_long=32768`); the engine starts SGLang with the flags above (bf16,
	`--mem-fraction-static 0.85`) on a free local port and stops it when the run ends. Use `--option gpu=<n>` to pick a
	GPU, and `--option sglang_python=<python>` if SGLang lives in another environment. `--option header_when_long=0` gives
	the Gen1 prompts.
	- `--option backend=hf`: transformers only, bf16, no server; slower, and not the setting of our runs.
	- Limits: up to 64 questions per request and 255 options per question. Larger requests are reported as
	unsupported; nothing is truncated.
	- Temperature in both backends: transformers and SGLang both read the final norm weight and
	`final_logit_softcapping` from these files, so both backends apply the folded temperature; the per-type temperatures
	of `temperatures.json` are applied by the front end, the same for both. Our Decision Index run and
	calibration numbers use SGLang.

	## Acknowledgements

	- Read-twice layout and question header: FlexDM ([github.com/1000luca/FlexDM](https://github.com/1000luca/FlexDM), MIT
	licence).
	- Question header: also used by Aleria Judge ([github.com/ALERIA-ML/aleria-judge](https://github.com/ALERIA-ML/aleria-judge), Apache-2.0).
	- Input repetition: Xu et al., 2023, "Re-Reading Improves Reasoning in Large Language Models"
	([arXiv:2309.06275](https://arxiv.org/abs/2309.06275)).