Qwen3.5-4B-3D-Coder · ONNX int4 (CPU)

First ONNX export of ccore/Qwen3.5-4B-3D-Coder (4B, Qwen3.5 hybrid attention: 24 gated-delta linear layers + 8 full-attention layers).

17 GB fp32 → 2.8 GB int4, runs on CPU with onnxruntime alone, 21 tok/s generation on a laptop (M1 Pro, no GPU, no Metal). Weights are 4-bit MatMulNBits (block 32, symmetric, accuracy_level=4); activations stay fp32.

Measured (M1 Pro, CPU only, `run_text.py`)


decode	21.1 tok/s (13.8 tok/s in the tighter verification harness)
prefill	2.7 s for a 20-token chat prompt
weights	2.84 GB (prefill) + 2.64 GB (decode)
token embedding	636 MB int8 (max abs err 7.6e-4)
first-token argmax	identical to fp32 (token 1596)
greedy divergence	from token 5 of 32

Quality, stated honestly: greedy decoding at 4 bits diverges from fp32 within a few tokens — that is normal and not a defect. Same prompt, both runs:

int4 : We need answer likely code. Need write function merges two sorted lists. Need perhaps
       explain. Need ensure robust. Use two pointers, compare, append. Handle empty
fp32 : We need answer likely provide code. Need write function merges two sorted lists. Need
       perhaps explain. Need ensure no hidden issue. User asks straightforward. We can provide

Same intent, same structure, different wording from token 5. Compare with logit-level or task-level metrics before trusting token identity as a quality signal.

Files

file	what
`text_prefill.onnx` (+`.data`)	prefill / teacher-forcing, 256-token windows, KV+recurrent states as I/O
`text_decode.onnx` (+`.data`)	one token per call, same state layout
`emb_i8.npy`, `emb_i8_scale.npy`	token embedding, int8 with per-row scale
`run_text.py`	runner: numpy + onnxruntime (transformers only for the tokenizer)
`config.json`, `tokenizer*`, `chat_template.jinja`	metadata

Usage

pip install onnxruntime numpy transformers
python run_text.py --prompt "Write a Python function that merges two sorted lists." --max-new 128

from run_text import Coder
from transformers import AutoTokenizer
m, tok = Coder(), AutoTokenizer.from_pretrained("ccore/Qwen3.5-4B-3D-Coder")
text, ids, stats = m("Explain what a monad is.", 128, tok)
print(stats)   # {'prefill_s': 2.7, 'decode_tok_s': 21.1, ...}

The graphs take inputs_embeds (not token ids), so the embedding table ships separately. State tensors (64 of them: conv_*/rec_* for the 24 linear layers, k_*/v_* for the 8 attention layers) are inputs and outputs — run_text.py derives their shapes from the graph itself.

How it was built (and what bit us)

Export in fp32, quantize int4. fp16 activations are tempting (half the file) but this model's forward upcasts to fp32 internally (.float() in norms, fp32 rope/decay scalars), so the exported graph ends up fp16-with-fp32-islands and ORT rejects every op whose operands disagree — one node at a time. fp32 activations + fp32 scales is the combination MatMulNBits is built around, and it loads first try.
Export and quantize in separate processes. A 4B fp32 graph peaks near 17 GB; quantizing in the same interpreter that still holds the export OOMs a 32 GB machine.
Pre-transpose Linear weights before export. MatMulNBitsQuantizer silently quantizes nothing on MatMul(x, Transpose(W)) and on Gemm — the file comes out the same size and bit-identical. Swapping each nn.Linear for one holding a [in, out] weight makes the weight a plain constant initializer, which is the form it quantizes.
States, not a kv-cache. The gated-delta layers carry a conv buffer plus a recurrent matrix; treating the cache as key/value pairs only works for 8 of the 32 layers.
onnx.save(save_as_external_data=True) appends to an existing .data file rather than truncating it, so saving twice doubles the file. Write to a fresh path, then swap.

Limits

This is the text path. The base repo is a VLM (it has a vision tower for 3D/code-from-image); that tower is not in this release.
CPU only in the measurements above; the graphs are plain ONNX so CUDA/DirectML/CoreML EPs will load them, but no speed numbers are claimed for those.
4-bit weights, fp32 activations: ~7 GB peak RSS at runtime.