| --- |
| license: apache-2.0 |
| base_model: PelaAI/KnowLine-4B-Gen5.5 |
| library_name: onnxruntime |
| pipeline_tag: text-classification |
| tags: |
| - onnx |
| - decision-model |
| - qwen3.5 |
| --- |
| |
| # KnowLine-4B-Gen5.5 — ONNX |
|
|
| ONNX builds of `/home/anon/LLM/KnowLine-4B-Gen5.5` |
| (PelaAI KnowLine-4B-Gen5.5 = Qwen3.5-4B fine-tune, `Qwen3_5ForConditionalGeneration`), |
| runnable with the latest onnxruntime. No torch needed at inference. |
|
|
| This is a **decision model**, not a chatbot: you send a state plus typed |
| questions (yes/no, pick-one, scored rubric) and get back a probability |
| distribution per question — one prefill each, no text generation. The main |
| script is `score_knowline.py`; `generate.py` (free text) is secondary. |
|
|
| ## Decision scoring |
|
|
| ```bash |
| pip install -r requirements.txt # GPU line, or swap to the CPU line |
| python score_knowline.py --request examples/support.json --model ./model-cuda-int4 --cuda |
| ``` |
|
|
| A request is a `/v1/systemone` body: `{"state": ..., "questions": {...}}`. |
| Three question types: |
|
|
| - `noul` — yes/no (`Should the agent offer a refund?` → `{"noul": 0.40}`) |
| - `choice` — pick one of k (`priority` → `urgent` at 0.88) |
| - `score` — rubric levels 0..k-1, returns expectation + full distribution |
|
|
| ### Example 1: support triage (`examples/support.json`) |
|
|
| State: customer whose laptop shipping stalled, promised callback never came, |
| needs it for work Monday. Four questions, one request: |
|
|
| ```bash |
| python score_knowline.py --request examples/support.json --model ./model-cuda-int4 --cuda |
| ``` |
|
|
| ```json |
| { |
| "answers": { |
| "refund": {"type": "noul", "noul": 0.5312}, |
| "priority": {"type": "choice", "choice": "urgent", |
| "probabilities": {"low": 0.052, "normal": 0.105, "urgent": 0.842}}, |
| "tone": {"type": "choice", "choice": "angry", |
| "probabilities": {"angry": 0.948, "neutral": 0.050, "happy": 0.002}}, |
| "satisfaction": {"type": "score", "score": 0.401, |
| "probabilities": {"0": 0.680, "1": 0.254, "2": 0.053, "3": 0.010, "4": 0.003}, |
| "legend": {"0": "very dissatisfied", "1": "dissatisfied", "2": "neutral", |
| "3": "satisfied", "4": "very satisfied"}} |
| }, |
| "usage": {"input_tokens": 1196, "output_tokens": 4} |
| } |
| ``` |
|
|
| Read: refund is a coin flip leaning yes (0.53 — escalate rather than |
| deciding either way), ticket is urgent, customer is angry, expected |
| satisfaction if handled normally is 0.40/4. |
|
|
| ### Example 2: content moderation (`examples/moderation.json`) |
|
|
| State: 2-day-old account posting a crypto "secret system" fee scam. Three |
| questions: |
|
|
| ```bash |
| python score_knowline.py --request examples/moderation.json --model ./model-cuda-int4 --cuda |
| ``` |
|
|
| ```json |
| { |
| "answers": { |
| "violates": {"type": "noul", "noul": 0.9987}, |
| "category": {"type": "choice", "choice": "scam", |
| "probabilities": {"legit": 0.0004, "spam": 0.006, "scam": 0.993, "abuse": 0.001}}, |
| "severity": {"type": "score", "score": 3.62, |
| "probabilities": {"0": 0.004, "1": 0.006, "2": 0.039, "3": 0.261, "4": 0.689}} |
| }, |
| "usage": {"input_tokens": 1028, "output_tokens": 3} |
| } |
| ``` |
|
|
| Read: violates policy (0.999), it's a scam not mere spam (0.993), severity |
| 3.62/4 — auto-remove territory. High confidence on category (0.97), middling |
| on exact severity level (0.51): act on the category, have a human confirm |
| severe-vs-critical. |
|
|
| ### Notes for decision use |
|
|
| - Prompts use the Gen5.5 serving layout (chat template, read-twice on by |
| default, `--no-read-twice` for Gen5 prompts) and the label-token softmax, so |
| outputs are comparable with the PyTorch/SGLang serving. No calibration file |
| needed: T = 1.50 is folded into the weights. |
| - `score` returns the full distribution, not just the mean — use the spread |
| (e.g. severity 3-vs-4 mass) to route to humans, not just the expectation. |
| - Up to 64 questions per request, 255 options per question. Text only |
| (same limit as `--backend hf`; images need SGLang in the source repo). |
| - If your use depends on probability thresholds, re-check calibration on your |
| own data (see the model card); INT4 shifts probabilities slightly — measured |
| drift below. |
|
|
| ## Chat generation (secondary) |
|
|
| ```bash |
| python generate.py --model ./model-cuda-int4 --cuda --prompt "What is the capital of France?" |
| python generate.py --model ./model-cpu --interactive |
| ``` |
|
|
| ## Contents |
|
|
| | path | what | |
| |---|---| |
| | `model-cuda-int8/` | CUDA int8 build (most faithful): `model.onnx` + `model.onnx.data` (~5.0 GB). Full int8 weights, chunks of 32, running math untouched | |
| | `model-cuda-int4/` | CUDA int4 build: `model.onnx` + `model.onnx.data` (~2.7 GB). True int4, k-quant, block 32, only the LM head at int8 | |
| | `model-cpu-int8/` | CPU int8 build: `model.onnx` + `model.onnx.data` (~6.9 GB). Same recipe as the CUDA int8, with 8-bit running math | |
| | `model-cpu/` | CPU int4 build: `model.onnx` + `model.onnx.data` (~2.6 GB) | |
| | `examples/` | ready-made decision requests (`support.json`, `moderation.json`) | |
| | `onnx_raw.py` | shared runner: `OnnxLM` (prefill / step / generate) | |
| | `generate.py` | chat / text generation | |
| | `score_knowline.py` | decision scoring (this README's main script) | |
| | `verify_parity.py` | torch-bf16 vs ONNX scoring comparison (needs torch) | |
| | `compare_all.py` | accuracy + speed check of every build vs the original (needs torch) | |
| | `knowline_server.py`, `knowline_layout.py` | copies from the source repo (MIT) used for rendering | |
| | `orig_tokenizer/` | pristine tokenizer — scoring prompts stay byte-identical to PyTorch serving | |
| | `fix_tokenizer.py` | tokenizer fix for the GenAI C++ tokenizer (applied to all four builds) | |
| | `requirements.txt` | runtime deps (pinned latest versions) | |
| | `export*.log`, `parity*.log`, `fp16_cpu_session_check.log` | build + verification logs | |
| | `config.json` (in each build) | source architecture config (Hub convention) | |
| | `.gitattributes` | LFS tracking for `*.onnx*` (Hub convention) | |
| | `.gitignore` | ignores `__pycache__/` etc. | |
| | `LICENSE` | MIT for the runner scripts (weights Apache-2.0, serving files MIT — see below) | |
|
|
| ## How it was built |
|
|
| Plain `torch.onnx.export` / optimum cannot handle this model: 24 of the 32 |
| decoder layers are GatedDeltaNet linear-attention (recurrent + conv state), |
| which has no standard ONNX representation. The supported path is Microsoft's |
| **onnxruntime-genai ModelBuilder**, which supports Qwen3.5 (see the official |
| `microsoft/olive-recipes` example `Qwen-Qwen3.5-4B/builtin`). |
|
|
| Export (onnxruntime-genai-cuda 0.19.0, onnxruntime-gpu 1.31.1, |
| transformers 5.22.1, torch 2.13; logs in `export*.log`): |
|
|
| ```bash |
| # CUDA int8 (most faithful): full int8 weights, chunks of 32, running math |
| # untouched, smarter packing |
| python -m onnxruntime_genai.models.builder \ |
| -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cuda-int8 -p int8 -e cuda \ |
| --extra_options block_size=32 algo_config=k_quant |
| |
| # CUDA int4: true int4, k-quant, block 32, only the LM head at int8 |
| python -m onnxruntime_genai.models.builder \ |
| -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cuda-int4 -p int4 -e cuda \ |
| --extra_options block_size=32 algo_config=k_quant \ |
| matmul_mixed_precision=last_matmul:int8 |
| |
| # CPU int4, for machines without CUDA |
| python -m onnxruntime_genai.models.builder \ |
| -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cpu -p int4 -e cpu \ |
| --extra_options accuracy_level=4 block_size=32 |
| |
| # CPU int8 (most faithful on CPU): int8 weights + smarter packing |
| python -m onnxruntime_genai.models.builder \ |
| -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cpu-int8 -p int8 -e cpu \ |
| --extra_options accuracy_level=4 block_size=32 algo_config=k_quant |
| |
| python fix_tokenizer.py model-cuda-int8 model-cuda-int4 model-cpu-int8 model-cpu # C++ std::regex compat, per Olive recipe |
| ``` |
|
|
| All builds are **text-only decoders** (`qwen3_5_text`); the vision tower is not |
| exported. |
|
|
| ## Running it: raw onnxruntime (not onnxruntime-genai) |
|
|
| The runners drive the graph with **plain onnxruntime** (`onnx_raw.py`: |
| prefill + greedy decode with manual KV/conv/recurrent state). Reason: |
| onnxruntime-genai 0.19.0 fails at `Model` load for this export on CPU with |
| `GetMemoryInfo for CPU should not be used` (its Qwen3.5 recurrent-state setup), |
| while onnxruntime 1.31.1 runs the identical graph fine. `genai_config.json` |
| is kept as exported, so the GenAI path can be retried with future releases. |
|
|
| ## Verification |
|
|
| Measured on this machine (RTX 3090 Ti graphics card + processor) with |
| `compare_all.py`: the original (full precision, graphics card) against each |
| build on the same 4 requests / 10 questions. Times are per request, after |
| warmup. |
|
|
| | model | file size | support request | moderation request | biggest miss vs original | same answers | |
| |---|---|---|---|---|---| |
| | original (full precision) | ~8.8 GB | 0.36 s | 0.29 s | — | — | |
| | `model-cuda-int8` | 5.0 GB | 0.61 s | 0.50 s | 0.019 | 10 of 10 | |
| | `model-cuda-int4` | 2.7 GB | 0.60 s | 0.49 s | 0.093 | 9 of 10 | |
| | `model-cpu-int8` | 6.9 GB | 5.8 s | 4.9 s | 0.023 | 10 of 10 | |
| | `model-cpu` | 2.6 GB | 6.5 s | 5.4 s | 0.172 | 10 of 10 | |
|
|
| - The 8-bit builds are the faithful ones: worst miss 0.019 on graphics, |
| 0.023 on processor. The coin-flip refund lands 0.43 on both vs 0.44 |
| original, same answer (no) — the processor 8-bit fixes the old 4-bit |
| processor's 0.27 wobble on that question. |
| - The 4-bit graphics build's one weak spot: the support refund question |
| reads **0.53 vs 0.44 original** — a coin-flip question landing across 0.5. |
| Treat near-0.5 scores as "escalate", not as decisions. |
| - The 4-bit processor build gets every answer right but its numbers are the |
| softest (worst miss 0.17); prefer the 8-bit processor build wherever |
| thresholds matter. |
| - "Capital of France?" → `Paris` on the graphics-card builds (first word in |
| ~0.2 s on the 8-bit, peak 4.4 GB graphics memory). |
| - Weight files checked present and size-consistent in all four builds |
| (8-bit weights, 5.0 GB; 710 4-bit graphics weights, 2.7 GB; |
| 8-bit processor weights, 6.9 GB; 557 4-bit processor weights, 2.6 GB). |
|
|
| ## Notes / limits |
|
|
| - Provider matrix is strict — each build runs only on its target EP: |
| `model-cuda-int8` and `model-cuda-int4` on CUDA (`--cuda`), `model-cpu` on CPU. |
| The CUDA exports use the `GatedDeltaNet` operator (no CPU kernel exists); |
| the CPU export uses `LinearAttention` instead. Cross combinations fail at |
| load with an explicit message. Verified: all four builds live-run (CUDA on a |
| 3090 Ti, CPU on CPU). |
| - MTP draft heads (`mtp.onnx*`) were deleted from all four builds: |
| nothing here uses speculative decoding — the runners load only `model.onnx`. |
| Re-running the export commands above regenerates them. |
| - The small builds shift probabilities slightly vs full precision. Measured |
| worst miss: 0.019 (graphics 8-bit), 0.093 (graphics 4-bit), |
| 0.023 (processor 8-bit), 0.172 (processor 4-bit). |
| - On `accuracy_level`: 0 = activations untouched (most faithful, the CUDA |
| default — used for the CUDA int4 build); 4 = activations quantized to int8 |
| (the CPU default). Higher number does not mean more accurate. |
| - onnxruntime-genai's C++ tokenizer uses the fixed `tokenizer.json` in each |
| build dir; scoring tokenizes with `orig_tokenizer/` instead, so prompts are |
| unaffected by that fix. |
|
|
| ## License |
|
|
| - Weights (`*.onnx.data`): Apache-2.0, same as the base model (Qwen3.5-4B). |
| - `knowline_server.py` / `knowline_layout.py`: MIT, PelaAI (byte-identical |
| copies of the source repo files). |
| - Runner scripts (`onnx_raw.py`, `generate.py`, `score_knowline.py`, |
| `verify_parity.py`, `fix_tokenizer.py`): MIT — see `LICENSE`. |
|
|
