happyme531/muscriptor-medium-onnx

MuScriptor Medium — optimized W4A32 ONNX Runtime

This is an unofficial derivative ONNX conversion of MuScriptor/muscriptor-medium, developed by Mirelo × Kyutai. It is not affiliated with or endorsed by the MuScriptor authors.

The conversion uses upstream revision f32236969308476e01fd3aae67357de5feb05a2d. The source checkpoint SHA-256 is ac80adbdf85d87231735fd948af7013441c0afced316c4e9067fd5d8a7fb97ec.

Model variant

The deployable model is in onnx/w4a32_optimized.

Component	Storage / graph dtype
Transformer backbone	symmetric RTN INT4, group size 32
Graph activations and KV cache	FP32
Token and metadata embeddings	FP32, not quantized
LM head	FP32, not quantized
Conditioner	FP32, not quantized

Only the 96 constant-weight MatMul operations in the 24-layer Transformer backbone are converted to ONNX Runtime MatMulNBits. The graph uses accuracy_level=4: its public tensors remain FP32, while supported CPU kernels dynamically quantize MatMul activations to INT8 and accumulate in INT32.

The decoder additionally contains 24 GroupQueryAttention and 48 SkipLayerNormalization nodes. Its fixed-capacity KV cache supports ONNX Runtime past/present buffer sharing through I/O Binding, avoiding allocation and copying of the complete cache on every token.

decoder.onnx.data contains external tensor data and must remain next to decoder.onnx.

Quick smoke test

hf download happyme531/muscriptor-medium-onnx --local-dir muscriptor-medium-onnx
cd muscriptor-medium-onnx
uv venv
source .venv/bin/activate
uv pip install -r runtime/requirements.txt
python runtime/test_onnxruntime.py \
  --model-dir onnx/w4a32_optimized \
  --provider cpu --max-new-tokens 8 --intra-op-threads 16

Use --audio path/to/audio.wav to replace the deterministic five-second 440 Hz smoke-test tone. The helper validates low-level conditioner/decoder execution and greedy token generation; it is not the complete upstream multi-chunk audio-to-MIDI application.

The default cache length is 2504 positions. A shared FP32 cache at that length occupies about 469.5 MiB. Lower --max-cache-length when the deployment has a smaller generation budget.

Validation

The model passes onnx.checker.check_model and was executed with ONNX Runtime 1.26.0 on x86-64 CPU. Results below use a Xeon Gold 6278C, one NUMA node, 16 intra-op threads, shared KV buffers, a five-second 440 Hz input, and a 576-position cache:

Metric	Result
Prefill	468.0 ms
Steady decode, mean over 56 measured tokens	7.74 ms/token
Steady decode median	7.73 ms/token
Steady decode p90	7.85 ms/token

A 64-step teacher-forced comparison against the original FP32 PyTorch model gave 81.25% top-1 agreement, 0.741 mean top-5 overlap, 0.286 centered relative logit L2 error, and 0.208 mean probability total variation. This synthetic smoke input is useful for regression testing, not a transcription-quality benchmark.

ARM64 execution was not measured on the target SoC. ONNX Runtime's CPU kernels provide an ARM64 path for this graph: the quantized backbone can use MLAS NEON dot-product kernels, while GQA's FP32 matrix multiplications use MLAS SGEMM. KleidiAI/SME acceleration additionally depends on the ONNX Runtime build and the device's runtime CPU features.

Files

File	Size	SHA-256
onnx/w4a32_optimized/conditioner.onnx	6,223,112 bytes	c1ea7895ade9760538f42f63110a782bcc1606dee89f2ee434db2251434a3c7d
onnx/w4a32_optimized/decoder.onnx	79,814 bytes	74e1181288fa5205754e9661cbccafc5182d035beadd4412da1ecd00515c3de9
onnx/w4a32_optimized/decoder.onnx.data	217,354,240 bytes	4bc6ac2807854632bacc0ef9b5e9e1871545e573194fe7fc197948730aa3c8ef

Total model storage, excluding the small JSON config, is 223,657,166 bytes (213.30 MiB).

License, attribution, and conditions

The original model weights and these derivative ONNX weights are distributed under Creative Commons Attribution-NonCommercial 4.0 International. Commercial use is not permitted under that license. Users must also comply with the upstream supplemental conditions, including holding all necessary rights to input audio and complying with applicable laws.

See NOTICE.md for attribution and UPSTREAM_MODEL_CARD.md for the original model card, intended uses, limitations, supplemental conditions, author list, and citation. The runtime helpers are covered by their MIT license.