electroglyph/KnowLine-4B-Gen5.5-onnx

KnowLine-4B-Gen5.5 — ONNX

ONNX builds of /home/anon/LLM/KnowLine-4B-Gen5.5 (PelaAI KnowLine-4B-Gen5.5 = Qwen3.5-4B fine-tune, Qwen3_5ForConditionalGeneration), runnable with the latest onnxruntime. No torch needed at inference.

This is a decision model, not a chatbot: you send a state plus typed questions (yes/no, pick-one, scored rubric) and get back a probability distribution per question — one prefill each, no text generation. The main script is score_knowline.py; generate.py (free text) is secondary.

Decision scoring

pip install -r requirements.txt          # GPU line, or swap to the CPU line
python score_knowline.py --request examples/support.json --model ./model-cuda-int4 --cuda

A request is a /v1/systemone body: {"state": ..., "questions": {...}}. Three question types:

• noul — yes/no (Should the agent offer a refund? → {"noul": 0.40})
• choice — pick one of k (priority → urgent at 0.88)
• score — rubric levels 0..k-1, returns expectation + full distribution

Example 1: support triage (examples/support.json)

State: customer whose laptop shipping stalled, promised callback never came, needs it for work Monday. Four questions, one request:

python score_knowline.py --request examples/support.json --model ./model-cuda-int4 --cuda

{
  "answers": {
    "refund": {"type": "noul", "noul": 0.5312},
    "priority": {"type": "choice", "choice": "urgent",
      "probabilities": {"low": 0.052, "normal": 0.105, "urgent": 0.842}},
    "tone": {"type": "choice", "choice": "angry",
      "probabilities": {"angry": 0.948, "neutral": 0.050, "happy": 0.002}},
    "satisfaction": {"type": "score", "score": 0.401,
      "probabilities": {"0": 0.680, "1": 0.254, "2": 0.053, "3": 0.010, "4": 0.003},
      "legend": {"0": "very dissatisfied", "1": "dissatisfied", "2": "neutral",
                 "3": "satisfied", "4": "very satisfied"}}
  },
  "usage": {"input_tokens": 1196, "output_tokens": 4}
}

Read: refund is a coin flip leaning yes (0.53 — escalate rather than deciding either way), ticket is urgent, customer is angry, expected satisfaction if handled normally is 0.40/4.

Example 2: content moderation (examples/moderation.json)

State: 2-day-old account posting a crypto "secret system" fee scam. Three questions:

python score_knowline.py --request examples/moderation.json --model ./model-cuda-int4 --cuda

{
  "answers": {
    "violates": {"type": "noul", "noul": 0.9987},
    "category": {"type": "choice", "choice": "scam",
      "probabilities": {"legit": 0.0004, "spam": 0.006, "scam": 0.993, "abuse": 0.001}},
    "severity": {"type": "score", "score": 3.62,
      "probabilities": {"0": 0.004, "1": 0.006, "2": 0.039, "3": 0.261, "4": 0.689}}
  },
  "usage": {"input_tokens": 1028, "output_tokens": 3}
}

Read: violates policy (0.999), it's a scam not mere spam (0.993), severity 3.62/4 — auto-remove territory. High confidence on category (0.97), middling on exact severity level (0.51): act on the category, have a human confirm severe-vs-critical.

Notes for decision use

• Prompts use the Gen5.5 serving layout (chat template, read-twice on by default, --no-read-twice for Gen5 prompts) and the label-token softmax, so outputs are comparable with the PyTorch/SGLang serving. No calibration file needed: T = 1.50 is folded into the weights.
• score returns the full distribution, not just the mean — use the spread (e.g. severity 3-vs-4 mass) to route to humans, not just the expectation.
• Up to 64 questions per request, 255 options per question. Text only (same limit as --backend hf; images need SGLang in the source repo).
• If your use depends on probability thresholds, re-check calibration on your own data (see the model card); INT4 shifts probabilities slightly — measured drift below.

Chat generation (secondary)

python generate.py --model ./model-cuda-int4 --cuda --prompt "What is the capital of France?"
python generate.py --model ./model-cpu --interactive

Contents

path	what
model-cuda-int8/	CUDA int8 build (most faithful): model.onnx + model.onnx.data (~5.0 GB). Full int8 weights, chunks of 32, running math untouched
model-cuda-int4/	CUDA int4 build: model.onnx + model.onnx.data (~2.7 GB). True int4, k-quant, block 32, only the LM head at int8
model-cpu-int8/	CPU int8 build: model.onnx + model.onnx.data (~6.9 GB). Same recipe as the CUDA int8, with 8-bit running math
model-cpu/	CPU int4 build: model.onnx + model.onnx.data (~2.6 GB)
examples/	ready-made decision requests (support.json, moderation.json)
onnx_raw.py	shared runner: OnnxLM (prefill / step / generate)
generate.py	chat / text generation
score_knowline.py	decision scoring (this README's main script)
verify_parity.py	torch-bf16 vs ONNX scoring comparison (needs torch)
compare_all.py	accuracy + speed check of every build vs the original (needs torch)
knowline_server.py, knowline_layout.py	copies from the source repo (MIT) used for rendering
orig_tokenizer/	pristine tokenizer — scoring prompts stay byte-identical to PyTorch serving
fix_tokenizer.py	tokenizer fix for the GenAI C++ tokenizer (applied to all four builds)
requirements.txt	runtime deps (pinned latest versions)
export*.log, parity*.log, fp16_cpu_session_check.log	build + verification logs
config.json (in each build)	source architecture config (Hub convention)
.gitattributes	LFS tracking for *.onnx* (Hub convention)
.gitignore	ignores __pycache__/ etc.
LICENSE	MIT for the runner scripts (weights Apache-2.0, serving files MIT — see below)

How it was built

Plain torch.onnx.export / optimum cannot handle this model: 24 of the 32 decoder layers are GatedDeltaNet linear-attention (recurrent + conv state), which has no standard ONNX representation. The supported path is Microsoft's onnxruntime-genai ModelBuilder, which supports Qwen3.5 (see the official microsoft/olive-recipes example Qwen-Qwen3.5-4B/builtin).

Export (onnxruntime-genai-cuda 0.19.0, onnxruntime-gpu 1.31.1, transformers 5.22.1, torch 2.13; logs in export*.log):

# CUDA int8 (most faithful): full int8 weights, chunks of 32, running math
# untouched, smarter packing
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cuda-int8 -p int8 -e cuda \
  --extra_options block_size=32 algo_config=k_quant

# CUDA int4: true int4, k-quant, block 32, only the LM head at int8
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cuda-int4 -p int4 -e cuda \
  --extra_options block_size=32 algo_config=k_quant \
  matmul_mixed_precision=last_matmul:int8

# CPU int4, for machines without CUDA
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cpu -p int4 -e cpu \
  --extra_options accuracy_level=4 block_size=32

# CPU int8 (most faithful on CPU): int8 weights + smarter packing
python -m onnxruntime_genai.models.builder \
  -i /home/anon/LLM/KnowLine-4B-Gen5.5 -o model-cpu-int8 -p int8 -e cpu \
  --extra_options accuracy_level=4 block_size=32 algo_config=k_quant

python fix_tokenizer.py model-cuda-int8 model-cuda-int4 model-cpu-int8 model-cpu   # C++ std::regex compat, per Olive recipe

All builds are text-only decoders (qwen3_5_text); the vision tower is not exported.

Running it: raw onnxruntime (not onnxruntime-genai)

The runners drive the graph with plain onnxruntime (onnx_raw.py: prefill + greedy decode with manual KV/conv/recurrent state). Reason: onnxruntime-genai 0.19.0 fails at Model load for this export on CPU with GetMemoryInfo for CPU should not be used (its Qwen3.5 recurrent-state setup), while onnxruntime 1.31.1 runs the identical graph fine. genai_config.json is kept as exported, so the GenAI path can be retried with future releases.

Verification

Measured on this machine (RTX 3090 Ti graphics card + processor) with compare_all.py: the original (full precision, graphics card) against each build on the same 4 requests / 10 questions. Times are per request, after warmup.

model	file size	support request	moderation request	biggest miss vs original	same answers
original (full precision)	~8.8 GB	0.36 s	0.29 s	—	—
model-cuda-int8	5.0 GB	0.61 s	0.50 s	0.019	10 of 10
model-cuda-int4	2.7 GB	0.60 s	0.49 s	0.093	9 of 10
model-cpu-int8	6.9 GB	5.8 s	4.9 s	0.023	10 of 10
model-cpu	2.6 GB	6.5 s	5.4 s	0.172	10 of 10

• The 8-bit builds are the faithful ones: worst miss 0.019 on graphics, 0.023 on processor. The coin-flip refund lands 0.43 on both vs 0.44 original, same answer (no) — the processor 8-bit fixes the old 4-bit processor's 0.27 wobble on that question.
• The 4-bit graphics build's one weak spot: the support refund question reads 0.53 vs 0.44 original — a coin-flip question landing across 0.5. Treat near-0.5 scores as "escalate", not as decisions.
• The 4-bit processor build gets every answer right but its numbers are the softest (worst miss 0.17); prefer the 8-bit processor build wherever thresholds matter.
• "Capital of France?" → Paris on the graphics-card builds (first word in ~0.2 s on the 8-bit, peak 4.4 GB graphics memory).
• Weight files checked present and size-consistent in all four builds (8-bit weights, 5.0 GB; 710 4-bit graphics weights, 2.7 GB; 8-bit processor weights, 6.9 GB; 557 4-bit processor weights, 2.6 GB).

Notes / limits

• Provider matrix is strict — each build runs only on its target EP: model-cuda-int8 and model-cuda-int4 on CUDA (--cuda), model-cpu on CPU. The CUDA exports use the GatedDeltaNet operator (no CPU kernel exists); the CPU export uses LinearAttention instead. Cross combinations fail at load with an explicit message. Verified: all four builds live-run (CUDA on a 3090 Ti, CPU on CPU).
• MTP draft heads (mtp.onnx*) were deleted from all four builds: nothing here uses speculative decoding — the runners load only model.onnx. Re-running the export commands above regenerates them.
• The small builds shift probabilities slightly vs full precision. Measured worst miss: 0.019 (graphics 8-bit), 0.093 (graphics 4-bit), 0.023 (processor 8-bit), 0.172 (processor 4-bit).
• On accuracy_level: 0 = activations untouched (most faithful, the CUDA default — used for the CUDA int4 build); 4 = activations quantized to int8 (the CPU default). Higher number does not mean more accurate.
• onnxruntime-genai's C++ tokenizer uses the fixed tokenizer.json in each build dir; scoring tokenizes with orig_tokenizer/ instead, so prompts are unaffected by that fix.

License

• Weights (*.onnx.data): Apache-2.0, same as the base model (Qwen3.5-4B).
• knowline_server.py / knowline_layout.py: MIT, PelaAI (byte-identical copies of the source repo files).
• Runner scripts (onnx_raw.py, generate.py, score_knowline.py, verify_parity.py, fix_tokenizer.py): MIT — see LICENSE.