INFERENCE.md

KnowLine-31B-Gen1.5: serving and Decision Index reproduction

Gen1.5 is KnowLine-31B-Gen1 with a new way of serving long requests: the weights are the same files, requests that fit the read-twice limit are read twice as in Gen1, and longer text-only requests get a question header (see Question header for long requests). These settings reproduce our self-run Decision Index 0.3 evaluation of Gen1.5, scored 2026-10-28 19:22 CST. That run was not sent fresh in full: the requests whose prompts are the same under the Gen1 and Gen1.5 front ends were carried over from our KnowLine-31B-Gen1 run, and the requests that change were sent fresh through the Gen1.5 release front end; see Decision Index run.

Weights

• The weights are byte-identical to KnowLine-31B-Gen1: every weight, config and tokenizer file has the same SHA-256 as there (SHA256SUMS). They were made in three steps (bf16, about 63 GB):
  1. Fine-tuning: gemma-4-31B-it trained for one epoch with a LoRA adapter (rank 32, alpha 64, language model only), merged into the weights.
  2. Averaging with the base model: every tensor is 0.75 × the fine-tuned tensor + 0.25 × the gemma-4-31B-it tensor (computed in fp32, stored in bf16). Tensors that fine-tuning did not change, such as the vision tower, stay those of the base model.
  3. Readout temperature T = 1.44, folded in two places so that every final logit is divided exactly by T:
     • the final norm weight model.language_model.norm.weight is divided by 1.44 (Gemma 4's RMSNorm applies the weight as x * w, not (1 + w) * x);
     • text_config.final_logit_softcapping in config.json is 20.833333333333336 instead of 30.0. Gemma 4 caps its logits as cap * tanh(z / cap); with both changes the output is 30 * tanh(z / 30) / T, the capped logit divided by T. Do not change the soft-cap back to 30, or the temperature is only partly applied.
     • T was fitted by negative log-likelihood on our in-training evaluation sets, served read twice through the Gen1 front end (4,965 questions); refitting on the released weights gives T = 1.00, and ECE on our held-out sets (6,995 questions) is 0.044 (0.102 at T = 1 before folding).
• The architecture is Gemma4ForConditionalGeneration. Apart from final_logit_softcapping, config.json is identical to that of google/gemma-4-31B-it; the tokenizer files, the chat template, processor_config.json and generation_config.json are byte-identical to the base model.
• SHA256SUMS lists every file. Only the serving files (knowline_server.py, knowline_engine.py, serve_knowline.sh) and the documentation differ from Gen1; knowline_layout.py and temperatures.json are the same.

Software

component	version
SGLang	0.5.21 (torch 2.13.0, CUDA 13.0)
transformers	5.12.1
front end	knowline_server.py + knowline_layout.py (this repository; need only transformers and requests)
decision-index kit	0.3 at commit 62d2f51de34a2de64906345b6bc3e98e27ff55c7

About the front end

knowline_server.py is the /v1/systemone front end of our runs; the prompt layout lives in knowline_layout.py next to it. It is the Gen1 front end with one addition, the question header for long requests (--header-when-long). Requests that read twice covers, and requests with images, are rendered exactly as by the Gen1 front end, prompt for prompt. It renders with the model's own chat template. The front end contains:

• rendering: the chat template with thinking off; the state is followed by one user turn with the instruction, the question and all options labelled A, B, ...; the assistant turn opens with Answer:;
• read twice (on by default, see below);
• the question header for text-only requests that read twice does not cover (on in serve_knowline.sh, see below);
• label-token scoring: one prefill per question, then a softmax over the option labels only;
• the HTTP server.

Read twice

• What it does: for a text-only request of at most 3,000 tokens (counted with the model's tokenizer over the whole request read once), every turn after the system turn is written a second time before the answer is read: the state, the instruction, the question and its options. The second copy starts with the line "Read the same input again before answering:". The answer is still read from the label token after Answer:, one prefill per question.
• What it leaves alone: requests over 3,000 tokens are not repeated: they get the question header (next section) or, past its limit, are sent once as in Gen1. Requests with images are sent once. In our Decision Index run, 132,376 of the 140,620 requests (94.1%) were read twice, the same requests as in the Gen1 run.
• Switches: --no-read-twice reads every request once, without the question header; --twice-max-tokens sets the cut. knowline_engine.py takes --option read_twice=0 and --option twice_max_tokens=<n> for the same. All our evaluations of this model used read twice.
• Cost: about 1.54 times the prompt tokens of reading once for the requests it covers (over the whole suite, 323 million against 209 million with the Gen1 serving, the same rows and tokenizer as KnowLine-26B-A4B-Gen2.5's run); see Hardware and latency.
• Credit: the read-twice layout follows FlexDM (github.com/1000luca/FlexDM, MIT), including its "Read the same input again before answering:" line; repeating the input goes back to RE2 (Xu et al., 2023, arXiv:2309.06275). knowline_layout.py is our own implementation.

Question header for long requests

• What it does: for a text-only request that read twice does not cover (more than 3,000 tokens read once), a header goes before the state: the line "Questions that will be asked about the conversation or state below:", then one line "- " per question of the request, in request order and without the options. The request is then read once; the instruction, the question block, the options and Answer: are as in Gen1, and the answer is read from the label token, one prefill per question. Requests with a single question get the header too.
• Limit: the header is used only if the longest of the request's prompts with it is at most 32,768 tokens (counted with the model's tokenizer); otherwise the request keeps the Gen1 read-once prompts.
• What it leaves alone: requests that read twice covers and requests with images are byte-identical to Gen1. In our Decision Index run, 7,845 of the 140,620 requests (5.6%) got the header, and 399 (0.3%) were over the limit and were sent as in Gen1.
• Switches: serve_knowline.sh passes --header-when-long 32768. knowline_server.py started on its own defaults to --header-when-long 0, which gives exactly the Gen1 prompts. knowline_engine.py uses 32768 by default; --option header_when_long=0 turns it off.
• Cost: the header is part of every question's prompt, and requests with many questions have long headers. Over the whole suite the front end's prompt tokens, summed over questions, are 896 million, against 323 million with the Gen1 serving and 209 million reading every request once. With SGLang's prefix cache the header and the state are prefilled once per request; counted that way over all suite requests, the prefilled tokens are about 5% more than with the Gen1 serving.
• Credit: the question listing follows FlexDM (github.com/1000luca/FlexDM, MIT), with "conversation or state" in place of its "context". Aleria Judge (github.com/ALERIA-ML/aleria-judge, Apache-2.0) uses the same header structure.

The front end uses temperature 1 and the readout temperature is in the weights, so both backends get it. As in Gen1, temperatures.json ({"score": 0.6944}, shipped in this repository) makes the front end divide the logits of score (rubric) questions by a further 0.6944, so they are read at an effective temperature of 1.44 × 0.6944 = 1.0, while yes/no and choice questions stay at 1.44. Score questions are often read as an expected value over the scale, which a higher temperature pulls toward the middle. serve_knowline.sh passes the file and knowline_engine.py loads it by default (--option temperatures=none turns it off). KnowLine releases before Gen1 did not use per-type temperatures. All our evaluations of this model used them, except where noted.

Serve

serve_knowline.sh starts both processes, with the settings of KnowLine-31B-Gen1 plus --header-when-long 32768.

1. SGLang engine, bf16.
   • Served in bf16, without --quantization fp8, as the 26B-A4B (for which two FP8 runs of the same questions agreed on only 95.7% of the decisions, against 99.35% in bf16). We did not test FP8 at load for this model.
   • There is no --mamba-radix-cache-strategy flag (Gemma 4 has no recurrent layers).
   • SGLang reserves --mem-fraction-static (default 0.85) of the GPU's memory for the weights and its KV cache: about 82 GB on a 96 GB card, of which the weights take about 63 GB. The value only sizes the KV cache; scores do not depend on it. Every run below used one 96 GB H20 per server; we have not run it on smaller GPUs.
   • SGLang treats the model as multimodal by itself. The Decision Index run sent text only.
2. /v1/systemone front end: knowline_server.py, chat style, temperature 1, 16 scoring threads, read twice up to 3,000 tokens, the question header for longer text-only requests whose prompts fit 32,768 tokens (--header-when-long 32768), per-type temperatures from temperatures.json (score questions / 0.6944), box questions with --box-prompt gemma --box-coords yxyx (Gemma's box_2d format, y first) and box_calibration.json when present.

pip install "sglang==0.5.21" "transformers==5.12.1" requests   # torch / CUDA per SGLang's install docs
bash serve_knowline.sh PelaAI/KnowLine-31B-Gen1.5 0 8080          # GPU 0; SGLang on :9080, /v1/systemone on :8080
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

Equivalent manual launch, with the exact flags of our run:

CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path PelaAI/KnowLine-31B-Gen1.5 --served-model-name m \
  --tp 1 --mem-fraction-static 0.85 --enable-fp32-lm-head --port 9080 &
python knowline_server.py --model PelaAI/KnowLine-31B-Gen1.5 --backend sglang --url http://127.0.0.1:9080 \
  --served-model-name m --temperature 1 --workers 16 --port 8080 --temperatures temperatures.json \
  --header-when-long 32768 --box-prompt gemma --box-coords yxyx          # read twice is on by default

Without SGLang (through transformers; text only, no prefix cache, bf16):

pip install torch "transformers==5.12.1" requests
python knowline_server.py --model PelaAI/KnowLine-31B-Gen1.5 --backend hf --port 8080 --temperatures temperatures.json \
  --header-when-long 32768

• Loaded in bf16, the weights take about 62.5 GB (58.3 GiB) of GPU memory.
• Checked on one H20 with the Gen1 front end (read twice on, no question header); we did not repeat this check for Gen1.5: on 30 random Decision Index 0.3 rows, 80 of the 82 answers were the same as in our SGLang run; the other two are near-ties (the top two options within 0.06 of each other in one of the two runs). Requests took 576.5 ms median one at a time.
• Images need the SGLang backend.

Decision Index run

• Suite rows: the 140,620 rows that decision-index run --edition 0.3 sends: selected-rows.jsonl.gz, added-rows.jsonl.gz and gsm8k-rows.jsonl.gz of suite-0.3, with ToolRet, BRIGHT and ACOS cut to their scoring subsets.
• Carried over (132,376 rows): every request that read twice covers has the same prompts under the Gen1 and Gen1.5 front ends (the new code path runs only for requests that read twice does not cover; a byte-for-byte check of the rendered prompts on a random sample of requests of every kind found no difference). Their results were taken from our KnowLine-31B-Gen1 run (same weights, same SGLang flags, scored 2026-10-24 04:28 CST), unchanged.
• Sent fresh (8,244 rows): every request that read twice does not cover, sent through this repository's serve_knowline.sh on four identical servers, one per H20 (--mem-fraction-static 0.90, which only sizes the KV cache, as in the Gen1 run). Each part is a resumable decision-index run against its server. None failed and nothing was retried.
• The two groups were concatenated and scored with the kit: all 140,178 scoreable requests were answered; none was unsupported or truncated, and none failed.
• Consistency check: 830 rows (2,808 questions), sampled by benchmark from both groups, were sent again through the full Gen1.5 front end: 2,801 of the 2,808 questions (99.75%) got the same top answer as in the scored results (carried rows 1,283 of 1,289, fresh rows 1,518 of 1,519).

decision-index run --edition 0.3 --engine http --option base_url=http://127.0.0.1:<port> --option model=m --no-verify \
  --compact --rows parts/<i>.jsonl.gz --out parts/<i>          # the 8,244 rows, run in parallel
cat <carried results.jsonl> parts/*/results.jsonl > results.jsonl
decision-index score --suite-dir suite-0.3 --results results.jsonl --engine http --out score

Hardware and latency

• Four NVIDIA H20 (96 GB each), one server per GPU.
• Latency: we sent 700 Decision Index 0.3 rows, sampled in the style of the board's latency-v1 method (per-catalog counts, a systematic sample by payload length), one at a time to one server on one H20 with the release settings (after 10 warm-up rows).
  • Requests took 330.3 ms median, 1,135.1 ms mean, 1,417.7 ms at the 80th percentile and 4,240.9 ms at the 95th.
  • 1,602 of the 1,605 answers were the same as in our Decision Index results.
  • The Gen1 card's latency (340.3 ms median, 910.0 ms mean) and KnowLine-26B-A4B-Gen2.5's (66.3 ms median, 151.1 ms mean) were measured on a different sample of 300 rows, so they are not directly comparable.
• Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1).

One-command Decision Index run

knowline_engine.py (in this repository) is a Decision Index engine. It runs the same front end in process, so there is no server to start by hand:

pip install "sglang==0.5.21" "transformers==5.12.1" requests        # plus the decision-index kit
git clone https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 && cd KnowLine-31B-Gen1.5   # puts the three files on the path
python -m decision_index pipeline --engine knowline_engine:KnowLine \
    --option model=PelaAI/KnowLine-31B-Gen1.5 --option revision=<commit> --out runs/KnowLine-31B-Gen1.5

• Default backend and layout (the setting of our runs): read twice is on and the question header is on for longer requests (header_when_long=32768); the engine starts SGLang with the flags above (bf16, --mem-fraction-static 0.85) on a free local port and stops it when the run ends. Use --option gpu=<n> to pick a GPU, and --option sglang_python=<python> if SGLang lives in another environment. --option header_when_long=0 gives the Gen1 prompts.
• --option backend=hf: transformers only, bf16, no server; slower, and not the setting of our runs.
• Limits: up to 64 questions per request and 255 options per question. Larger requests are reported as unsupported; nothing is truncated.
• Temperature in both backends: transformers and SGLang both read the final norm weight and final_logit_softcapping from these files, so both backends apply the folded temperature; the per-type temperatures of temperatures.json are applied by the front end, the same for both. Our Decision Index run and calibration numbers use SGLang.

Acknowledgements

• Read-twice layout and question header: FlexDM (github.com/1000luca/FlexDM, MIT licence).
• Question header: also used by Aleria Judge (github.com/ALERIA-ML/aleria-judge, Apache-2.0).
• Input repetition: Xu et al., 2023, "Re-Reading Improves Reasoning in Large Language Models" (arXiv:2309.06275).