---
license: apache-2.0
base_model: PelaAI/KnowLine-4B-Gen5.5
library_name: onnxruntime
pipeline_tag: text-classification
tags:
- onnx
- decision-model
- qwen3.5
---

# KnowLine-4B-Gen5.5 — ONNX

ONNX builds of `/home/anon/LLM/KnowLine-4B-Gen5.5`
(PelaAI KnowLine-4B-Gen5.5 = Qwen3.5-4B fine-tune, `Qwen3_5ForConditionalGeneration`),
runnable with the latest onnxruntime. No torch needed at inference.

This is a **decision model**, not a chatbot: you send a state plus typed
questions (yes/no, pick-one, scored rubric) and get back a probability
distribution per question — one prefill each, no text generation. The main
script is `score_knowline.py`; `generate.py` (free text) is secondary.

## Decision scoring

```bash
pip install -r requirements.txt          # GPU line, or swap to the CPU line
python score_knowline.py --request examples/support.json --model ./model-cuda-int4 --cuda
```

A request is a `/v1/systemone` body: `{"state": ..., "questions": {...}}`.
Three question types:

- `noul` — yes/no (`Should the agent offer a refund?` → `{"noul": 0.40}`)
- `choice` — pick one of k (`priority` → `urgent` at 0.88)
- `score` — rubric levels 0..k-1, returns expectation + full distribution

### Example 1: support triage (`examples/support.json`)

State: customer whose laptop shipping stalled, promised callback never came,
needs it for work Monday. Four questions, one request:

```bash
python score_knowline.py --request examples/support.json --model ./model-cuda-int4 --cuda
```

```json
{
  "answers": {
    "refund": {"type": "noul", "noul": 0.5312},
    "priority": {"type": "choice", "choice": "urgent",
      "probabilities": {"low": 0.052, "normal": 0.105, "urgent": 0.842}},
    "tone": {"type": "choice", "choice": "angry",
      "probabilities": {"angry": 0.948, "neutral": 0.050, "happy": 0.002}},
    "satisfaction": {"type": "score", "score": 0.401,
      "probabilities": {"0": 0.680, "1": 0.254, "2": 0.053, "3": 0.010, "4": 0.003},
      "legend": {"0": "very dissatisfied", "1": "dissatisfied", "2": "neutral",
                 "3": "satisfied", "4": "very satisfied"}}
  },
  "usage": {"input_tokens": 1196, "output_tokens": 4}
}
```

Read: refund is a coin flip leaning yes (0.53 — escalate rather than
deciding either way), ticket is urgent, customer is angry, expected
satisfaction if handled normally is 0.40/4.

### Example 2: content moderation (`examples/moderation.json`)

State: 2-day-old account posting a crypto "secret system" fee scam. Three
questions:

```bash
python score_knowline.py --request examples/moderation.json --model ./model-cuda-int4 --cuda
```

```json
{
  "answers": {
    "violates": {"type": "noul", "noul": 0.9987},
    "category": {"type": "choice", "choice": "scam",
      "probabilities": {"legit": 0.0004, "spam": 0.006, "scam": 0.993, "abuse": 0.001}},
    "severity": {"type": "score", "score": 3.62,
      "probabilities": {"0": 0.004, "1": 0.006, "2": 0.039, "3": 0.261, "4": 0.689}}
  },
  "usage": {"input_tokens": 1028, "output_tokens": 3}
}
```

Read: violates policy (0.999), it's a scam not mere spam (0.993), severity
3.62/4 — auto-remove territory. High confidence on category (0.97), middling
on exact severity level (0.51): act on the category, have a human confirm
severe-vs-critical.

### Notes for decision use

- Prompts use the Gen5.5 serving layout (chat template, read-twice on by
  default, `--no-read-twice` for Gen5 prompts) and the label-token softmax, so
  outputs are comparable with the PyTorch/SGLang serving. No calibration file
  needed: T = 1.50 is folded into the weights.
- `score` returns the full distribution, not just the mean — use the spread
  (e.g. severity 3-vs-4 mass) to route to humans, not just the expectation.
- Up to 64 questions per request, 255 options per question. Text only
  (same limit as `--backend hf`; images need SGLang in the source repo).
- If your use depends on probability thresholds, re-check calibration on your
  own data (see the model card); INT4 shifts probabilities slightly — measured
  drift below.

## Chat generation (secondary)

```bash
python generate.py --model ./model-cuda-int4 --cuda --prompt "What is the capital of France?"
python generate.py --model ./model-cpu --interactive
```

## Contents

| path | what |
|---|---|
| `model-cuda-int8/` | CUDA int8 build (most faithful): `model.onnx` + `model.onnx.data` (~5.0 GB). Full int8 weights, chunks of 32, running math untouched |
| `model-cuda-int4/` | CUDA int4 build: `model.onnx` + `model.onnx.data` (~2.7 GB). True int4, k-quant, block 32, only the LM head at int8 |
| `model-cpu-int8/` | CPU int8 build: `model.onnx` + `model.onnx.data` (~6.9 GB). Same recipe as the CUDA int8, with 8-bit running math |
| `model-cpu/` | CPU int4 build: `model.onnx` + `model.onnx.data` (~2.6 GB) |
| `examples/` | ready-made decision requests (`support.json`, `moderation.json`) |
| `onnx_raw.py` | shared runner: `OnnxLM` (prefill / step / generate) |
| `generate.py` | chat / text generation |
| `score_knowline.py` | decision scoring (this README's main script) |
| `verify_parity.py` | torch-bf16 vs ONNX scoring comparison (needs torch) |
| `compare_all.py` | accuracy + speed check of every build vs the original (needs torch) |
| `knowline_server.py`, `knowline_layout.py` | copies from the source repo (MIT) used for rendering |
| `orig_tokenizer/` | pristine tokenizer — scoring prompts stay byte-identical to PyTorch serving |
| `fix_tokenizer.py` | tokenizer fix for the GenAI C++ tokenizer (applied to all four builds) |
| `requirements.txt` | runtime deps (pinned latest versions) |
| `export*.log`, `parity*.log`, `fp16_cpu_session_check.log` | build + verification logs |
| `config.json` (in each build) | source architecture config (Hub convention) |
| `.gitattributes` | LFS tracking for `*.onnx*` (Hub convention) |
| `.gitignore` | ignores `__pycache__/` etc. |
| `LICENSE` | MIT for the runner scripts (weights Apache-2.0, serving files MIT — see below) |

## How it was built

Plain `torch.onnx.export` / optimum cannot handle this model: 24 of the 32
decoder layers are GatedDeltaNet linear-attention (recurrent + conv state),
which has no standard ONNX representation. The supported path is Microsoft's
**onnxruntime-genai ModelBuilder**, which supports Qwen3.5 (see the official
`microsoft/olive-recipes` example `Qwen-Qwen3.5-4B/builtin`).

Export (onnxruntime-genai-cuda 0.19.0, onnxruntime-gpu 1.31.1,
transformers 5.22.1, torch 2.13; logs in `export*.log`):

```bash
# CUDA int8 (most faithful): full int8 weights, chunks of 32, running math
# untouched, smarter packing
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cuda-int8 -p int8 -e cuda \
  --extra_options block_size=32 algo_config=k_quant

# CUDA int4: true int4, k-quant, block 32, only the LM head at int8
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cuda-int4 -p int4 -e cuda \
  --extra_options block_size=32 algo_config=k_quant \
  matmul_mixed_precision=last_matmul:int8

# CPU int4, for machines without CUDA
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cpu -p int4 -e cpu \
  --extra_options accuracy_level=4 block_size=32

# CPU int8 (most faithful on CPU): int8 weights + smarter packing
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cpu-int8 -p int8 -e cpu \
  --extra_options accuracy_level=4 block_size=32 algo_config=k_quant

python fix_tokenizer.py model-cuda-int8 model-cuda-int4 model-cpu-int8 model-cpu   # C++ std::regex compat, per Olive recipe
```

All builds are **text-only decoders** (`qwen3_5_text`); the vision tower is not
exported.

## Running it: raw onnxruntime (not onnxruntime-genai)

The runners drive the graph with **plain onnxruntime** (`onnx_raw.py`:
prefill + greedy decode with manual KV/conv/recurrent state). Reason:
onnxruntime-genai 0.19.0 fails at `Model` load for this export on CPU with
`GetMemoryInfo for CPU should not be used` (its Qwen3.5 recurrent-state setup),
while onnxruntime 1.31.1 runs the identical graph fine. `genai_config.json`
is kept as exported, so the GenAI path can be retried with future releases.

## Verification

Measured on this machine (RTX 3090 Ti graphics card + processor) with
`compare_all.py`: the original (full precision, graphics card) against each
build on the same 4 requests / 10 questions. Times are per request, after
warmup.

| model | file size | support request | moderation request | biggest miss vs original | same answers |
|---|---|---|---|---|---|
| original (full precision) | ~8.8 GB | 0.36 s | 0.29 s | — | — |
| `model-cuda-int8` | 5.0 GB | 0.61 s | 0.50 s | 0.019 | 10 of 10 |
| `model-cuda-int4` | 2.7 GB | 0.60 s | 0.49 s | 0.093 | 9 of 10 |
| `model-cpu-int8` | 6.9 GB | 5.8 s | 4.9 s | 0.023 | 10 of 10 |
| `model-cpu` | 2.6 GB | 6.5 s | 5.4 s | 0.172 | 10 of 10 |

- The 8-bit builds are the faithful ones: worst miss 0.019 on graphics,
  0.023 on processor. The coin-flip refund lands 0.43 on both vs 0.44
  original, same answer (no) — the processor 8-bit fixes the old 4-bit
  processor's 0.27 wobble on that question.
- The 4-bit graphics build's one weak spot: the support refund question
  reads **0.53 vs 0.44 original** — a coin-flip question landing across 0.5.
  Treat near-0.5 scores as "escalate", not as decisions.
- The 4-bit processor build gets every answer right but its numbers are the
  softest (worst miss 0.17); prefer the 8-bit processor build wherever
  thresholds matter.
- "Capital of France?" → `Paris` on the graphics-card builds (first word in
  ~0.2 s on the 8-bit, peak 4.4 GB graphics memory).
- Weight files checked present and size-consistent in all four builds
  (8-bit weights, 5.0 GB; 710 4-bit graphics weights, 2.7 GB;
  8-bit processor weights, 6.9 GB; 557 4-bit processor weights, 2.6 GB).

## Notes / limits

- Provider matrix is strict — each build runs only on its target EP:
  `model-cuda-int8` and `model-cuda-int4` on CUDA (`--cuda`), `model-cpu` on CPU.
  The CUDA exports use the `GatedDeltaNet` operator (no CPU kernel exists);
  the CPU export uses `LinearAttention` instead. Cross combinations fail at
  load with an explicit message. Verified: all four builds live-run (CUDA on a
  3090 Ti, CPU on CPU).
- MTP draft heads (`mtp.onnx*`) were deleted from all four builds:
  nothing here uses speculative decoding — the runners load only `model.onnx`.
  Re-running the export commands above regenerates them.
- The small builds shift probabilities slightly vs full precision. Measured
  worst miss: 0.019 (graphics 8-bit), 0.093 (graphics 4-bit),
  0.023 (processor 8-bit), 0.172 (processor 4-bit).
- On `accuracy_level`: 0 = activations untouched (most faithful, the CUDA
  default — used for the CUDA int4 build); 4 = activations quantized to int8
  (the CPU default). Higher number does not mean more accurate.
- onnxruntime-genai's C++ tokenizer uses the fixed `tokenizer.json` in each
  build dir; scoring tokenizes with `orig_tokenizer/` instead, so prompts are
  unaffected by that fix.

## License

- Weights (`*.onnx.data`): Apache-2.0, same as the base model (Qwen3.5-4B).
- `knowline_server.py` / `knowline_layout.py`: MIT, PelaAI (byte-identical
  copies of the source repo files).
- Runner scripts (`onnx_raw.py`, `generate.py`, `score_knowline.py`,
  `verify_parity.py`, `fix_tokenizer.py`): MIT — see `LICENSE`.