Retrieve0 commited on
Commit
4db26d2
·
verified ·
1 Parent(s): d721847

Add Qwen3-ASR 0.6B EXL3 decoders

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ decoder-exl3-3bpw/tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ decoder-exl3-4bpw/tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: Qwen/Qwen3-ASR-0.6B-hf
4
+ library_name: exllamav3
5
+ pipeline_tag: automatic-speech-recognition
6
+ tags:
7
+ - qwen3-asr
8
+ - exll3
9
+ - exllamav3
10
+ - quantized
11
+ - 3bpw
12
+ - 4bpw
13
+ ---
14
+
15
+ <p align="center">
16
+ <a href="https://github.com/lost0427/Qwen3-ASR-TensorRT"><img src="https://img.shields.io/badge/GitHub-source-181717?logo=github" alt="GitHub source"></a>
17
+ <a href="https://huggingface.co/Retrieve0/Qwen3-ASR-0.6B-EXL3-Decoder"><img src="https://img.shields.io/badge/Hugging_Face-model-yellow?logo=huggingface" alt="Hugging Face model"></a>
18
+ </p>
19
+
20
+ # Qwen3-ASR 0.6B EXL3 Decoder
21
+
22
+ ## English
23
+
24
+ EXL3 quantized text-decoder weights derived from [Qwen/Qwen3-ASR-0.6B-hf](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf). Use with a compatible [TensorRT encoder](https://huggingface.co/Retrieve0/Qwen3-ASR-0.6B-TensorRT-Encoder). The [Qwen3-ASR-TensorRT repository](https://github.com/lost0427/Qwen3-ASR-TensorRT) integrates both parts for transcription. These weights use the ExLlamaV3 format and are loaded by the project runtime.
25
+
26
+ ### Variants
27
+
28
+ | Variant | Use |
29
+ | --- | --- |
30
+ | `decoder-exl3-3bpw` | Lower memory use |
31
+ | `decoder-exl3-4bpw` | Higher-precision alternative |
32
+
33
+ Each variant includes EXL3 `model.safetensors`, tokenizer files, model and generation configuration, and quantization configuration.
34
+
35
+ ### Requirements
36
+
37
+ - NVIDIA GPU and CUDA
38
+ - ExLlamaV3 1.6.6 with compatible PyTorch and CUDA versions
39
+ - Reference environment: PyTorch 2.10.0, CUDA 12.8, Python 3.10–3.13
40
+ - The [Qwen3-ASR-TensorRT repository](https://github.com/lost0427/Qwen3-ASR-TensorRT), which includes the encoder/decoder integration
41
+
42
+ Windows is tested and supported. Linux should work with compatible CUDA, TensorRT, PyTorch, and ExLlamaV3 packages.
43
+
44
+ ### Usage
45
+
46
+ ```powershell
47
+ uv sync
48
+
49
+ uv run qwen3-asr --model Qwen/Qwen3-ASR-0.6B-hf transcribe .\audio.wav `
50
+ --backend tensorrt-exllama `
51
+ --decoder-dir .\decoder-exl3-3bpw `
52
+ --language Chinese
53
+ ```
54
+
55
+ ### License and citation
56
+
57
+ These weights are derived from `Qwen/Qwen3-ASR-0.6B-hf` and are published under Apache-2.0. Retain the upstream attribution and license. ExLlamaV3 and runtime dependencies have their own licenses. Please cite Qwen3-ASR, ExLlamaV3, and the runtime project.
58
+
59
+ ## 中文
60
+
61
+ 本仓库提供基于 [Qwen/Qwen3-ASR-0.6B-hf](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) 转换的 EXL3 量化文本解码器权重。使用时需配套兼容的 [TensorRT encoder](https://huggingface.co/Retrieve0/Qwen3-ASR-0.6B-TensorRT-Encoder)。[Qwen3-ASR-TensorRT 仓库](https://github.com/lost0427/Qwen3-ASR-TensorRT)集成了 encoder 和 decoder,可用于完整转写。本权重采用 ExLlamaV3 格式,由项目运行时加载。
62
+
63
+ ### 量化版本
64
+
65
+ | 版本 | 用途 |
66
+ | --- | --- |
67
+ | `decoder-exl3-3bpw` | 显存占用较低 |
68
+ | `decoder-exl3-4bpw` | 精度较高的替代版本 |
69
+
70
+ 每个版本包含 EXL3 `model.safetensors`、tokenizer、模型与生成配置以及量化配置。
71
+
72
+ ### 运行要求
73
+
74
+ - NVIDIA GPU 与 CUDA
75
+ - ExLlamaV3 1.6.6,以及相互兼容的 PyTorch 和 CUDA 版本
76
+ - 参考环境:PyTorch 2.10.0、CUDA 12.8、Python 3.10–3.13
77
+ - [Qwen3-ASR-TensorRT 仓库](https://github.com/lost0427/Qwen3-ASR-TensorRT),其中已集成 encoder 和 decoder
78
+
79
+ 项目已在 Windows 上测试并可用。Linux 环境在 CUDA、TensorRT、PyTorch 和 ExLlamaV3 软件包版本兼容时也可运行。
80
+
81
+ ### 使用
82
+
83
+ ```powershell
84
+ uv sync
85
+
86
+ uv run qwen3-asr --model Qwen/Qwen3-ASR-0.6B-hf transcribe .\audio.wav `
87
+ --backend tensorrt-exllama `
88
+ --decoder-dir .\decoder-exl3-3bpw `
89
+ --language Chinese
90
+ ```
91
+
92
+ ### 许可证与引用
93
+
94
+ 这些权重基于 `Qwen/Qwen3-ASR-0.6B-hf`,以 Apache-2.0 发布。请保留上游归属说明和许可证。ExLlamaV3 与运行时依赖各自遵循其许可证。请引用 Qwen3-ASR、ExLlamaV3 及运行时项目。
decoder-exl3-3bpw/config.json ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "attention_bias": false,
3
+ "attention_dropout": 0.0,
4
+ "bos_token_id": null,
5
+ "eos_token_id": [
6
+ 151643,
7
+ 151645
8
+ ],
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1024,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 3072,
14
+ "layer_types": [
15
+ "full_attention",
16
+ "full_attention",
17
+ "full_attention",
18
+ "full_attention",
19
+ "full_attention",
20
+ "full_attention",
21
+ "full_attention",
22
+ "full_attention",
23
+ "full_attention",
24
+ "full_attention",
25
+ "full_attention",
26
+ "full_attention",
27
+ "full_attention",
28
+ "full_attention",
29
+ "full_attention",
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention",
38
+ "full_attention",
39
+ "full_attention",
40
+ "full_attention",
41
+ "full_attention",
42
+ "full_attention"
43
+ ],
44
+ "max_position_embeddings": 65536,
45
+ "max_window_layers": 28,
46
+ "model_type": "qwen3",
47
+ "num_attention_heads": 16,
48
+ "num_hidden_layers": 28,
49
+ "num_key_value_heads": 8,
50
+ "pad_token_id": 151645,
51
+ "rms_norm_eps": 1e-06,
52
+ "rope_parameters": {
53
+ "rope_theta": 1000000,
54
+ "rope_type": "default"
55
+ },
56
+ "sliding_window": null,
57
+ "tie_word_embeddings": false,
58
+ "use_cache": true,
59
+ "use_sliding_window": false,
60
+ "vocab_size": 151936,
61
+ "architectures": [
62
+ "Qwen3ForCausalLM"
63
+ ],
64
+ "quantization_config": {
65
+ "quant_method": "exl3",
66
+ "version": "1.6.6",
67
+ "bits": 3.0,
68
+ "head_bits": 6,
69
+ "calibration": {
70
+ "rows": 250,
71
+ "cols": 2048
72
+ },
73
+ "out_scales": "auto",
74
+ "codebook": "mul1"
75
+ }
76
+ }
decoder-exl3-3bpw/generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_sample": false,
3
+ "eos_token_id": [
4
+ 151643,
5
+ 151645
6
+ ],
7
+ "max_new_tokens": 512,
8
+ "pad_token_id": 151645,
9
+ "transformers_version": "5.13.0.dev0"
10
+ }
decoder-exl3-3bpw/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:60c6363a772a38f4f6cb64f085c84c1232edd89bfc6276096ef924cc302b0c85
3
+ size 594798403
decoder-exl3-3bpw/quantization_config.json ADDED
The diff for this file is too large to render. See raw diff
 
decoder-exl3-3bpw/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe1fad59be22a41ee293363fcf95fdedbc7c93f3b49270b1d2e18bd1399a7a05
3
+ size 11429653
decoder-exl3-3bpw/tokenizer_config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "audio_bos_token": "<|audio_start|>",
4
+ "audio_eos_token": "<|audio_end|>",
5
+ "audio_token": "<|audio_pad|>",
6
+ "backend": "tokenizers",
7
+ "bos_token": null,
8
+ "clean_up_tokenization_spaces": false,
9
+ "eos_token": "<|im_end|>",
10
+ "errors": "replace",
11
+ "image_token": "<|image_pad|>",
12
+ "is_local": true,
13
+ "local_files_only": false,
14
+ "model_max_length": 131072,
15
+ "model_specific_special_tokens": {
16
+ "audio_bos_token": "<|audio_start|>",
17
+ "audio_eos_token": "<|audio_end|>",
18
+ "audio_token": "<|audio_pad|>",
19
+ "image_token": "<|image_pad|>",
20
+ "video_token": "<|video_pad|>",
21
+ "vision_bos_token": "<|vision_start|>",
22
+ "vision_eos_token": "<|vision_end|>"
23
+ },
24
+ "pad_token": "<|endoftext|>",
25
+ "processor_class": "Qwen3ASRProcessor",
26
+ "split_special_tokens": false,
27
+ "tokenizer_class": "Qwen2Tokenizer",
28
+ "unk_token": null,
29
+ "video_token": "<|video_pad|>",
30
+ "vision_bos_token": "<|vision_start|>",
31
+ "vision_eos_token": "<|vision_end|>"
32
+ }
decoder-exl3-4bpw/config.json ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "attention_bias": false,
3
+ "attention_dropout": 0.0,
4
+ "bos_token_id": null,
5
+ "eos_token_id": [
6
+ 151643,
7
+ 151645
8
+ ],
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1024,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 3072,
14
+ "layer_types": [
15
+ "full_attention",
16
+ "full_attention",
17
+ "full_attention",
18
+ "full_attention",
19
+ "full_attention",
20
+ "full_attention",
21
+ "full_attention",
22
+ "full_attention",
23
+ "full_attention",
24
+ "full_attention",
25
+ "full_attention",
26
+ "full_attention",
27
+ "full_attention",
28
+ "full_attention",
29
+ "full_attention",
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention",
38
+ "full_attention",
39
+ "full_attention",
40
+ "full_attention",
41
+ "full_attention",
42
+ "full_attention"
43
+ ],
44
+ "max_position_embeddings": 65536,
45
+ "max_window_layers": 28,
46
+ "model_type": "qwen3",
47
+ "num_attention_heads": 16,
48
+ "num_hidden_layers": 28,
49
+ "num_key_value_heads": 8,
50
+ "pad_token_id": 151645,
51
+ "rms_norm_eps": 1e-06,
52
+ "rope_parameters": {
53
+ "rope_theta": 1000000,
54
+ "rope_type": "default"
55
+ },
56
+ "sliding_window": null,
57
+ "tie_word_embeddings": false,
58
+ "use_cache": true,
59
+ "use_sliding_window": false,
60
+ "vocab_size": 151936,
61
+ "architectures": [
62
+ "Qwen3ForCausalLM"
63
+ ],
64
+ "quantization_config": {
65
+ "quant_method": "exl3",
66
+ "version": "1.6.6",
67
+ "bits": 4.0,
68
+ "head_bits": 6,
69
+ "calibration": {
70
+ "rows": 250,
71
+ "cols": 2048
72
+ },
73
+ "out_scales": "auto",
74
+ "codebook": "mul1"
75
+ }
76
+ }
decoder-exl3-4bpw/generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_sample": false,
3
+ "eos_token_id": [
4
+ 151643,
5
+ 151645
6
+ ],
7
+ "max_new_tokens": 512,
8
+ "pad_token_id": 151645,
9
+ "transformers_version": "5.13.0.dev0"
10
+ }
decoder-exl3-4bpw/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1588d86b03874de33f644ee3f8ec79104592a2aa7c785e1f9f9691ff308fc740
3
+ size 649848643
decoder-exl3-4bpw/quantization_config.json ADDED
The diff for this file is too large to render. See raw diff
 
decoder-exl3-4bpw/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fe1fad59be22a41ee293363fcf95fdedbc7c93f3b49270b1d2e18bd1399a7a05
3
+ size 11429653
decoder-exl3-4bpw/tokenizer_config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "audio_bos_token": "<|audio_start|>",
4
+ "audio_eos_token": "<|audio_end|>",
5
+ "audio_token": "<|audio_pad|>",
6
+ "backend": "tokenizers",
7
+ "bos_token": null,
8
+ "clean_up_tokenization_spaces": false,
9
+ "eos_token": "<|im_end|>",
10
+ "errors": "replace",
11
+ "image_token": "<|image_pad|>",
12
+ "is_local": true,
13
+ "local_files_only": false,
14
+ "model_max_length": 131072,
15
+ "model_specific_special_tokens": {
16
+ "audio_bos_token": "<|audio_start|>",
17
+ "audio_eos_token": "<|audio_end|>",
18
+ "audio_token": "<|audio_pad|>",
19
+ "image_token": "<|image_pad|>",
20
+ "video_token": "<|video_pad|>",
21
+ "vision_bos_token": "<|vision_start|>",
22
+ "vision_eos_token": "<|vision_end|>"
23
+ },
24
+ "pad_token": "<|endoftext|>",
25
+ "processor_class": "Qwen3ASRProcessor",
26
+ "split_special_tokens": false,
27
+ "tokenizer_class": "Qwen2Tokenizer",
28
+ "unk_token": null,
29
+ "video_token": "<|video_pad|>",
30
+ "vision_bos_token": "<|vision_start|>",
31
+ "vision_eos_token": "<|vision_end|>"
32
+ }