KnowLine-4B-Gen5 / INFERENCE.md
PEScn's picture
KnowLine-4B-Gen5
dba367c
|
Raw History Blame Contribute Delete
7.36 kB
# KnowLine-4B-Gen5: serving and Decision Index reproduction
These settings reproduce our self-run Decision Index 0.3 evaluation of these weights: the full 0.3 public suite sent
fresh (no rows carried over from another run), scored 2026-10-21 CST.
## Weights
- This repository holds bf16 weights: a LoRA fine-tune (r32, alpha 64) of Qwen3.5-4B, merged, with the readout
temperature folded in:
- **Readout temperature.** The final norm weight `model.language_model.norm.weight` is rewritten as
`w' = (1 + w) / 1.50 - 1`. The norm is zero-centred and the output head is tied to the embeddings, so this divides
every output logit by T = 1.50. No other tensor changes; the vision tower and the MTP head (with its own norm) are
the base model's.
- The architecture is `Qwen3_5ForConditionalGeneration`. `config.json`, the tokenizer files and the chat template are
byte-identical to Gen1-Gen4.
- `SHA256SUMS` lists every file.
## Software
| component | version |
|---|---|
| SGLang | 0.5.21 (torch 2.13.0, CUDA 13.0, flashinfer-python 0.6.18) |
| transformers | 5.12.1 |
| front end | `knowline_server.py` (this repository; needs only transformers and requests) |
| decision-index kit | 0.3 at commit 62d2f51de34a2de64906345b6bc3e98e27ff55c7 |
### About the front end
`knowline_server.py` is the `/v1/systemone` front end of our runs, packaged as one file. It is the same file as in
Gen4; only the model name in its docstring changed. It contains:
- rendering: the chat template with thinking off; the state is followed by one user turn with the instruction, the
question and all options labelled A, B, ...; the assistant turn opens with `Answer:`;
- label-token scoring: one prefill per question, then a softmax over the option labels only;
- the HTTP server.
The front end still uses temperature 1. The calibration temperature is in the weights, so no option or calibration file
is needed, and both backends get it.
The Gen5 run used `knowline_server.py` through `serve_knowline.sh`, the same serving as the Gen4 release.
## Serve
`serve_knowline.sh` starts both processes.
1. **SGLang engine.**
- FP8 is on at serving: weights are stored in bf16 and SGLang quantizes them on load.
- `--mem-fraction-static 0.72` only sizes SGLang's KV cache. Any value works and scores do not depend on it.
- SGLang treats the model as multimodal by itself. The Decision Index run sent text only.
2. **`/v1/systemone` front end:** `knowline_server.py`, `chat` style, temperature 1, 16 scoring threads, no per-type
calibration file.
```bash
pip install "sglang==0.5.21" "transformers==5.12.1" requests # torch / CUDA per SGLang's install docs
bash serve_knowline.sh PelaAI/KnowLine-4B-Gen5 0 8080 # GPU 0; SGLang on :9080, /v1/systemone on :8080
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
"model": "m",
"state": "Customer: my order arrived broken, I want my money back.",
"questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
"tone": {"type": "choice", "instructions": "Customer tone?",
"criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'
```
Equivalent manual launch, with the exact flags of our run:
```bash
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path PelaAI/KnowLine-4B-Gen5 --served-model-name m --tp 1 \
--quantization fp8 --mem-fraction-static 0.72 --mamba-radix-cache-strategy extra_buffer --enable-fp32-lm-head --port 9080 &
python knowline_server.py --model PelaAI/KnowLine-4B-Gen5 --backend sglang --url http://127.0.0.1:9080 \
--served-model-name m --temperature 1 --workers 16 --port 8080
```
Without SGLang (CPU or a single GPU through transformers; slower, no prefix cache, bf16 rather than FP8):
```bash
pip install torch "transformers==5.12.1" requests
python knowline_server.py --model PelaAI/KnowLine-4B-Gen5 --backend hf --port 8080
```
## Decision Index run
- Suite rows: the 140,620 rows that `decision-index run --edition 0.3` sends: `selected-rows.jsonl.gz`,
`added-rows.jsonl.gz` and `gsm8k-rows.jsonl.gz` of `suite-0.3`, with ToolRet, BRIGHT and ACOS cut to their scoring
subsets.
- Four identical servers ran on four GPUs, one each, with the settings above. The rows are split round-robin into 64
shards, 16 per server. Each shard is a resumable `decision-index run` against its server:
```bash
decision-index run --edition 0.3 --engine http --option base_url=http://127.0.0.1:<port> --option model=m --no-verify \
--compact --rows shards/<i>.jsonl.gz --out shards/<i> # i = 0..63, run in parallel; 16 shards per server
cat shards/*/results.jsonl > results.jsonl
decision-index score --suite-dir suite-0.3 --results results.jsonl --engine http --out score
```
## Hardware and latency
- Four NVIDIA H20 (96 GB each), one server per GPU.
- **Latency and repeatability:** we sent 300 random Decision Index 0.3 rows (477 questions) one at a time to one server
on one H20 (`serve_knowline.sh`, FP8 at load, after a warm-up).
- Requests took 31.6 ms median, 58.8 ms mean and 143.9 ms at the 95th percentile.
- 474 of the 477 answers were the same as in our published run.
- Without SGLang (`--backend hf`, bf16): on 30 random Decision Index 0.3 rows, all 82 answers were the same as in our
published run; requests took 201 ms median one at a time, and the model peaked at about 13.6 GB of GPU memory.
- Latency was not measured on the board's reference hardware (1x RTX PRO 6000, latency-v1).
## One-command Decision Index run
`knowline_engine.py` (in this repository) is a Decision Index engine. It runs the same front end in process, so there
is no server to start by hand:
```bash
pip install "sglang==0.5.21" "transformers==5.12.1" requests # plus the decision-index kit
git clone https://huggingface.co/PelaAI/KnowLine-4B-Gen5 && cd KnowLine-4B-Gen5 # puts both files on the path
python -m decision_index pipeline --engine knowline_engine:KnowLine \
--option model=PelaAI/KnowLine-4B-Gen5 --option revision=<commit> --out runs/KnowLine-4B-Gen5
```
- **Default backend (the setting of our runs):** the engine starts SGLang with the flags above (FP8 at load) on a free
local port and stops it when the run ends. Use `--option gpu=<n>` to pick a GPU, and
`--option sglang_python=<python>` if SGLang lives in another environment.
- **`--option backend=hf`:** transformers only, bf16, no server; slower, and not the setting of our runs.
- **Limits:** up to 64 questions per request and 255 options per question. Larger requests are reported as
unsupported; nothing is truncated.
- **Temperature in both backends:** transformers (`Qwen3_5RMSNorm`) and SGLang (`GemmaRMSNorm`) both apply the final
norm as `(1 + w) * x`, so both backends apply the folded temperature. Our Decision Index run and calibration numbers
use SGLang.
## Changes
- 2026-10-08 (also in Gen1-Gen4): `knowline_server.py` accepts chats whose roles the chat template rejects (for
example `customer` / `agent`): such a state is rendered as one user message, like any other structured state, instead
of the request failing. Any other unexpected error returns HTTP 500 instead of closing the connection. Requests that
worked before give the same answers.