--- license: apache-2.0 base_model: Qwen/Qwen3-ASR-1.7B-hf library_name: exllamav3 pipeline_tag: automatic-speech-recognition tags: - qwen3-asr - exll3 - exllamav3 - quantized - 3bpw - 4bpw ---

GitHub source Hugging Face model

# Qwen3-ASR 1.7B EXL3 Decoder ## English EXL3 quantized text-decoder weights derived from [Qwen/Qwen3-ASR-1.7B-hf](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf). Use with a compatible [TensorRT encoder](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-TensorRT-Encoder). The [Qwen3-ASR-TensorRT repository](https://github.com/lost0427/Qwen3-ASR-TensorRT) integrates the encoder and decoder for transcription. These weights use the ExLlamaV3 format and are loaded by the project runtime. ### Variants | Variant | Approximate size | Use | | --- | ---: | --- | | `decoder-exl3-3bpw` | 1.30 GiB | Legacy 3 bpw export, ExLlamaV3 default calibration | | `decoder-exl3-4bpw` | 1.46 GiB | Legacy 4 bpw export, ExLlamaV3 default calibration | | [`decoder-exl3-3bpw-train-clean-100`](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/tree/main/decoder-exl3-3bpw-train-clean-100) | 1.30 GiB | Separately calibrated 3 bpw export | | [`decoder-exl3-4bpw-train-clean-100`](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/tree/main/decoder-exl3-4bpw-train-clean-100) | 1.46 GiB | Separately calibrated 4 bpw export | Each variant includes EXL3 `model.safetensors`, tokenizer files, model and generation configuration, and quantization configuration. EXL3 uses quantized weight tensors alongside floating-point scale, normalization, and auxiliary tensors; the variants are identified by their EXL3 bitrate. ### Calibration The existing 3 bpw and 4 bpw files are legacy exports. They were quantized with ExLlamaV3's default calibration data: 250 rows of 2,048 tokens. The `decoder-exl3-3bpw-train-clean-100` and `decoder-exl3-4bpw-train-clean-100` directories contain the calibrated exports. Their calibration tensors contain 200 packed rows of 2,048 tokens, built from 1,603 paired audio requests and transcripts in LibriSpeech `train-clean-100`. No `test-clean` samples were used. Calibration changes quantization statistics; it is not fine-tuning. On all 2,620 LibriSpeech `test-clean` utterances, the calibrated 4 bpw variant measured 1.9108% WER and 205.11 ms median latency, compared with 1.9390% and 204.36 ms for the legacy 4 bpw file. The calibrated 3 bpw variant measured 2.1178% WER and 192.18 ms median, compared with 2.0256% and 189.06 ms for the legacy 3 bpw file; calibration did not improve the 3 bpw tradeoff. Each variant was run once; timings are not repeated-run averages. The `train-clean-100` variants are tuned for LibriSpeech-style English read speech. For general-purpose audio and Chinese ASR, the legacy decoders remain the default recommendation. ### Comparison with GGUF Q4_K ![Corpus WER and median latency](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/resolve/main/qwen3-1.7b-calibration-vs-gguf.png) ![Median latency by stage](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/resolve/main/qwen3-1.7b-stage-latency.png) All configurations use all 2,620 test-clean utterances. The EXL3 decoders are about 1.9–2.1× faster than GGUF Q4_K. The official `qwen-asr` Transformers backend reaches 1.8863% WER at 863.52 ms, about 4.2–4.6× slower than the best EXL3 variant. The chart labels give WER, median latency, and the speed ratio. ### Requirements - NVIDIA GPU and CUDA - ExLlamaV3 1.6.6 with compatible PyTorch and CUDA versions - Reference environment: PyTorch 2.10.0, CUDA 12.8, Python 3.10–3.13 - The [Qwen3-ASR-TensorRT repository](https://github.com/lost0427/Qwen3-ASR-TensorRT), which includes the encoder/decoder integration Windows is tested and supported. Linux should work with compatible CUDA, TensorRT, PyTorch, and ExLlamaV3 packages. ### Usage ```powershell uv sync uv run qwen3-asr --model Qwen/Qwen3-ASR-1.7B-hf transcribe .\audio.wav ` --backend tensorrt-exllama ` --decoder-dir .\decoder-exl3-4bpw ` --language Chinese ``` Start the HTTP API with the same backend: ```powershell uv run qwen3-asr --model Qwen/Qwen3-ASR-1.7B-hf serve ` --backend tensorrt-exllama ` --decoder-dir .\decoder-exl3-4bpw ``` ### License and citation These weights are derived from `Qwen/Qwen3-ASR-1.7B-hf` and are published under Apache-2.0. Retain the upstream attribution and license. ExLlamaV3 and runtime dependencies have their own licenses. Please cite Qwen3-ASR, ExLlamaV3, and the runtime project. ## 中文 本仓库提供基于 [Qwen/Qwen3-ASR-1.7B-hf](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) 转换的 EXL3 量化文本解码器权重。使用时需配套兼容的 [TensorRT encoder](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-TensorRT-Encoder)。[Qwen3-ASR-TensorRT 仓库](https://github.com/lost0427/Qwen3-ASR-TensorRT)已集成 encoder 和 decoder，可用于完整转写。本权重采用 ExLlamaV3 格式，由项目运行时加载。 ### 量化版本 | 版本 | 大小约 | 用途 | | --- | ---: | --- | | `decoder-exl3-3bpw` | 1.30 GiB | 旧版 3 bpw 权重，ExLlamaV3 默认校准 | | `decoder-exl3-4bpw` | 1.46 GiB | 旧版 4 bpw 权重，ExLlamaV3 默认校准 | | [`decoder-exl3-3bpw-train-clean-100`](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/tree/main/decoder-exl3-3bpw-train-clean-100) | 1.30 GiB | 单独校准的新版 3 bpw 权重 | | [`decoder-exl3-4bpw-train-clean-100`](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/tree/main/decoder-exl3-4bpw-train-clean-100) | 1.46 GiB | 单独校准的新版 4 bpw 权重 | 每个版本包含 EXL3 `model.safetensors`、tokenizer、模型与生成配置、量化配置。EXL3 权重由量化张量以及浮点缩放、归一化和辅助张量组成，版本以 EXL3 bpw 标识。 ### 校准数据 现有的 3 bpw 和 4 bpw 权重是旧版导出，使用 ExLlamaV3 默认校准数据量化：250 行、每行 2,048 个 token。 `decoder-exl3-3bpw-train-clean-100` 和 `decoder-exl3-4bpw-train-clean-100` 目录保存新版权重。校准张量由 LibriSpeech `train-clean-100` 的 1,603 组音频请求和转写文本打包为 200 行，每行 2,048 个 token。校准未使用 `test-clean` 样本。校准会影响量化统计，不会微调模型。 在全部 2,620 条 LibriSpeech `test-clean` 音频上，新版 4 bpw 的 WER 为 1.9108%，中位延迟为 205.11 ms；旧版 4 bpw 分别为 1.9390% 和 204.36 ms。新版 3 bpw 的 WER 为 2.1178%，中位延迟为 192.18 ms；旧版 3 bpw 分别为 2.0256% 和 189.06 ms，校准没有改善 3 bpw 的取舍。每个版本只运行了一次，延迟不是多轮平均值。 `train-clean-100` 版本针对 LibriSpeech 风格的英语朗读调优。通用音频和中文 ASR 仍默认推荐 legacy 版本。 ### 与 GGUF Q4_K 对比 ![速度与 WER](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/resolve/main/qwen3-1.7b-calibration-vs-gguf.png) ![分阶段延迟](https://huggingface.co/Retrieve0/Qwen3-ASR-1.7B-EXL3-Decoder/resolve/main/qwen3-1.7b-stage-latency.png) 全部配置均使用 2,620 条 test-clean。EXL3 约比 GGUF Q4_K 快 2.0–2.1 倍。官方 `qwen-asr` Transformers 后端的 WER 为 1.8863%，中位延迟为 863.52 ms，比最佳 EXL3 版本慢约 4.4 倍。图中标注了 WER、中位延迟和速度倍数。 ### 运行要求 - NVIDIA GPU 与 CUDA - ExLlamaV3 1.6.6，以及相互兼容的 PyTorch 和 CUDA 版本 - 参考环境：PyTorch 2.10.0、CUDA 12.8、Python 3.10–3.13 - [Qwen3-ASR-TensorRT 仓库](https://github.com/lost0427/Qwen3-ASR-TensorRT)，其中已集成 encoder 和 decoder 项目已在 Windows 上测试并可用。Linux 环境在 CUDA、TensorRT、PyTorch 和 ExLlamaV3 软件包版本兼容时也可运行。 ### 使用 ```powershell uv sync uv run qwen3-asr --model Qwen/Qwen3-ASR-1.7B-hf transcribe .\audio.wav ` --backend tensorrt-exllama ` --decoder-dir .\decoder-exl3-4bpw ` --language Chinese ``` 使用相同 backend 启动 HTTP API： ```powershell uv run qwen3-asr --model Qwen/Qwen3-ASR-1.7B-hf serve ` --backend tensorrt-exllama ` --decoder-dir .\decoder-exl3-4bpw ``` ### 许可证与引用 这些权重基于 `Qwen/Qwen3-ASR-1.7B-hf`，以 Apache-2.0 发布。请保留上游归属说明和许可证。ExLlamaV3 与运行时依赖各自遵循其许可证。请引用 Qwen3-ASR、ExLlamaV3 及运行时项目。