neavo/SenseVoiceSmall-ONNX-WebGPU-W8A32

SenseVoiceSmall ONNX WebGPU W8A32

Built directly from Alibaba's official full SenseVoiceSmall checkpoint. The official model implementation is used for FP32 ONNX export, followed by 8-bit weight-only quantization. Activations remain FP32. No pruning, distillation, or requantization of an existing INT8 model is performed.

本版本直接从阿里官方完整原始权重生成。 全部 917 个参数项严格加载成功，共 233,999,167 个参数。先使用官方模型代码导出 FP32 ONNX，再进行 W8A32 量化。该版本替换此前基于第三方 sherpa ONNX 导出的权重。

The model is an experimental candidate validated with native onnxruntime-node 1.31.1 on Windows / NVIDIA. Browser ONNX Runtime Web and other platforms have not been validated. This is not a Transformers pipeline checkpoint.

Files and integrity

• model.w8a32.onnx: self-contained model, 267,521,956 bytes.
• tokens.txt: matching official vocabulary, 25,055 entries.
• am.mvn: official CMVN statistics for feature normalization, from the same pinned checkpoint revision. Required by the audio frontend; these statistics are not embedded in the ONNX graph.
• SHA256SUMS: integrity hashes.
• LICENSE: upstream FunASR model license.
• configuration.json: ModelScope repository metadata.

Model SHA-256: e6e4badd5ef43c545e09a940ac519928f39aea9a09935d00fa3495ae324b0229.

Official provenance

Component	Pinned source
Original model	Alibaba / FunAudioLLM / FunASR SenseVoiceSmall
Checkpoint	iic/SenseVoiceSmall/model.pt
Checkpoint revision	70514a3da51f1160f51d18449dab6128bbd4928b
Checkpoint SHA-256	833ca2dcfdf8ec91bd4f31cfac36d6124e0c459074d5e909aec9cabe6204a3ea
Official implementation	FunAudioLLM/SenseVoice
Official code revision	586bbdb2b052f4fe19322581825228b6810a9fb3
Official vocabulary	tokens.json

Official config.yaml, am.mvn, and SentencePiece model define preprocessing and vocabulary semantics.

Export: official SenseVoiceSmall / export_meta.py, PyTorch 2.15.0+cpu, legacy ONNX exporter, opset 14, ONNX 1.23.2. Quantization: ONNX Runtime 1.31.1 MatMulNBitsQuantizer, constant MatMul weights only, bits=8, block_size=32, is_symmetric=True, accuracy_level=1, QOperator format. The full original checkpoint was loaded with strict=True. Quantized derivative prepared by neavo for LinguaGacha.

Input/output contract — changed from the previous release

This version preserves the official export names. Callers of the previous sherpa-derived version must update their feeds and output lookup:

Old name	Current name	Type / meaning
x	speech	float32, [batch, frames, 560] precomputed features
x_length	speech_lengths	int32, [batch], valid input frames
language	language	int32, [batch], language control ID
text_norm	textnorm	int32, [batch], text normalization control ID
First output	ctc_logits	float32, CTC logits
Not exposed	encoder_out_lens	Valid encoded lengths, including four control positions

Language IDs: auto 0, Chinese 3, English 4, Cantonese 7, Japanese 11, Korean 12. Text normalization: with ITN 14, without ITN 15.

The model consumes features, not raw audio. Use the official 16 kHz frontend, 80-bin filterbanks, LFR m=7, n=6, and the included am.mvn for CMVN. Decode CTC using tokens.txt and valid output lengths; the first four positions contain control information. The artifact does not include a full audio frontend or application pipeline.

import * as ort from 'onnxruntime-node'; // tested: 1.31.1

const session = await ort.InferenceSession.create('model.w8a32.onnx', {
  executionProviders: ['webgpu'],
  intraOpNumThreads: 4,
  interOpNumThreads: 1,
});
// const { ctc_logits, encoder_out_lens } = await session.run({
//   speech, speech_lengths, language, textnorm,
// });
await session.release();

For CPU use of this candidate, executionProviders: ['cpu'] and intraOpNumThreads: 1 avoid the multithreaded teardown problem in the local candidate checks, at a significant speed cost. The original dynamic INT8 model remains the better CPU-oriented choice.

Validation and performance

Environment: Windows x64, Intel i5-14600K, NVIDIA RTX 4070 Ti SUPER, native ONNX Runtime 1.31.1.

1. Official checkpoint: 917 parameter entries loaded strictly, no missing or unexpected entries.
2. Official PyTorch versus exported FP32 ONNX on two real Japanese feature inputs: maximum absolute logit errors about 2.48e-5 and 2.47e-4; encoded lengths matched exactly. ONNX validation passed.
3. GPU profiling: all 281 MatMulNBits and 140 remaining MatMul nodes ran on WebGPU. Seven host/device copy nodes were recorded; some shape-related operations still ran on CPU.
4. A broader check covered 64 English/Japanese windows, about 958 seconds. All outputs were finite. FP32 CPU and WebGPU token sequences matched in 64/64 cases. W8A32 WebGPU matched FP32 tokens in 47/64 cases. All three broader-check processes completed and exited normally.

Two-input timing test: one initial run per input, then five alternating timed repetitions; medians below. Inference timing includes transfers but excludes model loading, feature extraction, CTC decoding and file IO. CPU FP32 uses four threads; CPU W8A32 uses one.

Model / provider	10s input	20s input
Official-source FP32 / CPU	231.41 ms	423.66 ms
Official-source FP32 / WebGPU	43.42 ms	52.25 ms
Official-source W8A32 / WebGPU	39.67 ms	51.76 ms
Official-source W8A32 / CPU, one thread	1182.44 ms	1878.24 ms

In the 64-window check, total inference times were about 21.36s for FP32 CPU, 3.18s for FP32 WebGPU and 3.34s for W8A32 WebGPU. These are local measurements, not cross-platform speed guarantees.

Limitations

• Quantization is not lossless. Seventeen of 64 broader-check token sequences differed from FP32. This count is output agreement, not an error rate. No human-adjudicated multilingual WER/CER benchmark was performed.
• The separate 20s Japanese sample changed ディ社って to ィ社って under W8A32, on both CPU and WebGPU.
• Native process teardown is not guaranteed stable by these tests. The short FP32 CPU test completed inference and wrote results but timed out during exit; the broader FP32 CPU run exited normally. Earlier experiments also observed teardown stalls with WebGPU on other export graphs. Do not interpret this as a model-specific CPU-only defect or a proven upstream fix.
• Chinese, Korean, Cantonese, other GPUs, macOS/Linux, browsers, long-form audio and full application workflows remain unvalidated.

Attribution and license

Original SenseVoiceSmall model: Alibaba / FunAudioLLM / FunASR. Official model card, official implementation, paper.

The official Hugging Face model card identifies the FunASR Model Open Source License Agreement, included as LICENSE. Preserve the original source, author information and SenseVoiceSmall model name when redistributing. The model license is separate from the toolkit software license.