Retrieve0 commited on
Commit
7e911d9
·
verified ·
1 Parent(s): e1c4e48

Update benchmark numbers

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -40,7 +40,7 @@ The existing `decoder-exl3-3bpw` and `decoder-exl3-4bpw` files are legacy export
40
 
41
  The separately published `decoder-exl3-3bpw-train-clean-100` and `decoder-exl3-4bpw-train-clean-100` directories contain the calibrated exports. Their calibration tensors contain 200 packed rows of 2,048 tokens, built from 1,603 paired audio requests and transcripts in LibriSpeech `train-clean-100`. No `test-clean` samples were used. Calibration changes quantization statistics; it is not fine-tuning.
42
 
43
- On all 2,620 LibriSpeech `test-clean` utterances, the calibrated 3 bpw variant measured 2.5715% WER and 114.37 ms median latency, compared with 2.9029% and 115.17 ms for the legacy 3 bpw file. The calibrated 4 bpw variant measured 2.3776% WER and 115.05 ms median, compared with 2.4228% and 113.99 ms for the legacy 4 bpw file. These are two-pass 2026-11-10 rerun medians; timings are not repeated-run averages.
44
 
45
  The `train-clean-100` variants are tuned for LibriSpeech-style English read speech. For general-purpose audio and Chinese ASR, the legacy decoders remain the default recommendation.
46
 
@@ -50,7 +50,7 @@ The `train-clean-100` variants are tuned for LibriSpeech-style English read spee
50
 
51
  ![Median latency by stage](https://huggingface.co/Retrieve0/Qwen3-ASR-0.6B-EXL3-Decoder/resolve/main/qwen3-0.6b-stage-latency.png)
52
 
53
- All configurations use all 2,620 test-clean utterances. The EXL3 decoders are about 2.6× faster than GGUF Q4_K. The official `qwen-asr` Transformers backend reaches 2.3212% WER at 787.12 ms, about 6.8× slower than the best EXL3 variant. The chart labels give WER, median latency, and the speed ratio.
54
 
55
  ### Requirements
56
 
@@ -97,7 +97,7 @@ These weights are derived from `Qwen/Qwen3-ASR-0.6B-hf` and are published under
97
 
98
  独立目录 `decoder-exl3-3bpw-train-clean-100` 和 `decoder-exl3-4bpw-train-clean-100` 中保存新版权重。校准张量由 LibriSpeech `train-clean-100` 的 1,603 组音频请求和转写文本打包为 200 行,每行 2,048 个 token。校准未使用 `test-clean` 样本。校准会影响量化统计,不会微调模型。
99
 
100
- 在全部 2,620 条 LibriSpeech `test-clean` 音频上,新版 3 bpw 的 WER 为 2.5715%,中位延迟为 114.37 ms;旧版 3 bpw 分别为 2.9029% 和 115.17 ms。新版 4 bpw 的 WER 为 2.3776%,中位延迟为 115.05 ms;旧版 4 bpw 分别为 2.4228% 和 113.99 ms。以上为 2026-11-10 两轮复测的中位数,延迟不是多轮平均值。
101
 
102
  `train-clean-100` 版本针对 LibriSpeech 风格的英语朗读调优。通用音频和中文 ASR 仍默认推荐 legacy 版本。
103
 
 
40
 
41
  The separately published `decoder-exl3-3bpw-train-clean-100` and `decoder-exl3-4bpw-train-clean-100` directories contain the calibrated exports. Their calibration tensors contain 200 packed rows of 2,048 tokens, built from 1,603 paired audio requests and transcripts in LibriSpeech `train-clean-100`. No `test-clean` samples were used. Calibration changes quantization statistics; it is not fine-tuning.
42
 
43
+ On all 2,620 LibriSpeech `test-clean` utterances, the calibrated 3 bpw variant measured 2.5527% WER and 98.32 ms median latency, compared with 2.8784% and 103.43 ms for the legacy 3 bpw file. The calibrated 4 bpw variant measured 2.3419% WER and 96.83 ms median, compared with 2.4228% and 96.65 ms for the legacy 4 bpw file. These are two-pass 2026-11-10 rerun medians; timings are not repeated-run averages.
44
 
45
  The `train-clean-100` variants are tuned for LibriSpeech-style English read speech. For general-purpose audio and Chinese ASR, the legacy decoders remain the default recommendation.
46
 
 
50
 
51
  ![Median latency by stage](https://huggingface.co/Retrieve0/Qwen3-ASR-0.6B-EXL3-Decoder/resolve/main/qwen3-0.6b-stage-latency.png)
52
 
53
+ All configurations use all 2,620 test-clean utterances. The EXL3 decoders are about 2.9–3.1× faster than GGUF Q4_K. The official `qwen-asr` Transformers backend reaches 2.3212% WER at 787.12 ms, about 8.1× slower than the best EXL3 variant. The chart labels give WER, median latency, and the speed ratio.
54
 
55
  ### Requirements
56
 
 
97
 
98
  独立目录 `decoder-exl3-3bpw-train-clean-100` 和 `decoder-exl3-4bpw-train-clean-100` 中保存新版权重。校准张量由 LibriSpeech `train-clean-100` 的 1,603 组音频请求和转写文本打包为 200 行,每行 2,048 个 token。校准未使用 `test-clean` 样本。校准会影响量化统计,不会微调模型。
99
 
100
+ 在全部 2,620 条 LibriSpeech `test-clean` 音频上,新版 3 bpw 的 WER 为 2.5527%,中位延迟为 98.32 ms;旧版 3 bpw 分别为 2.8784% 和 103.43 ms。新版 4 bpw 的 WER 为 2.3419%,中位延迟为 96.83 ms;旧版 4 bpw 分别为 2.4228% 和 96.65 ms。以上为 2026-11-10 两轮复测的中位数,延迟不是多轮平均值。
101
 
102
  `train-clean-100` 版本针对 LibriSpeech 风格的英语朗读调优。通用音频和中文 ASR 仍默认推荐 legacy 版本。
103