Qwen3.8-Flash-Next, EXL3 packs for mlx-serve
Each branch of this repo holds one of turboderp's EXL3 quantizations of
Qwen3.8-Flash-Next, repacked into the file layout that
mlx-serve loads on Apple silicon. turboderp made them with
exllamav3 and publishes them in
turboderp/Qwen3.8-Flash-Next-exl3. The quantizations are
his and nothing is quantized again, so please credit turboderp when you use them. mlx-serve serves them with the
turboexl3_qwen38 model plugin, or with full-EXL3 support for Qwen3.8 and the turboexl3 quant plugin, the route the
packs were first measured on. The experts load into memory, while the n-gram table, 26 to 39 GB depending on the
branch, stays on SSD and is read per token, so it costs disk and file cache rather than memory.
Each bitrate is a branch, and main holds only this card and the license.
Branches
| Branch | bpw | Branch size | n-gram table, not in the branch | KLD / PPL (turboderp's) | Source and credit | Status |
|---|---|---|---|---|---|---|
5.05bpw |
5.05 | 84.38 GB | 39.04 GB | 0.0040 / 1.3848 | turboderp's 5.05bpw_h6_ng6 |
live |
4.05bpw |
4.05 | 68.59 GB | 39.04 GB | 0.0066 / 1.3897 | turboderp's 4.05bpw_h6_ng6 |
live |
3.05bpw |
3.05 | 52.72 GB | 32.64 GB | 0.0174 / 1.4074 | turboderp's 3.05bpw_h5_ng5 |
live |
2.05bpw |
2.05 | 36.85 GB | 26.24 GB | 0.0681 / 1.4871 | turboderp's 2.05bpw_h4_ng4 |
live |
Sizes are in GB, where 1 GB is 10^9 bytes. The branches do not include the n-gram table, and The n-gram table says where each one comes from. KLD and PPL are turboderp's, measured with exllamav3 on his files, as Quality explains. Each branch's card says what is in its pack, where each file comes from and how to run it with mlx-serve.
Related repositories
| Repo | What it holds |
|---|---|
| turboderp/Qwen3.8-Flash-Next-exl3 | turboderp's EXL3 quantizations, a branch per bitrate. It is the source of every branch here, and of the n-gram tables. |
qwen38/ in OpenSourceWTF/mlx-serve-streaming-pack-converters |
The conversion scripts that repack his files into mlx-serve's layout, including the n-gram table step for 2.05 and 3.05. They moved here from OpenSourceWTF/qwen38-exl3-mlx-serve-converter, which is archived. |
| OpensourceWTF/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve | The earlier repo of 5.05bpw, now a stub that points here. |
| OpensourceWTF/Qwen3.8-Flash-Next-EXL3-4.05-mlx-serve | The earlier repo of 4.05bpw, now a stub that points here. |
| OpensourceWTF/Qwen3.8-Flash-Next-EXL3-3.05-mlx-serve | The earlier repo of 3.05bpw, now a stub that points here. |
| OpensourceWTF/Qwen3.8-Flash-Next-EXL3-2.05-mlx-serve | The earlier repo of 2.05bpw, now a stub that points here. |
Each branch holds its earlier repo's files, without ngram_embedding.safetensors and with a new card.
How to download and run
Requirements
The packs can be served in two ways. The first is the turboexl3_qwen38 model plugin. The second is the route the packs were first measured on, and every speed table on this card comes from it.
We tested these packs on one Apple M5 Max with 128 GB of memory and did not test other Macs. We tested text only, not image or video input. The packs do not load in exllamav3, transformers, vLLM, SGLang or mlx-lm. For exllamav3, use turboderp's repository.
The turboexl3_qwen38 model plugin
The main branch of CommunityServe/plugins has the turboexl3_qwen38 model
plugin, which serves these four packs. Its source is in
plugins/turboexl3_qwen38. It claims a
qwen4_exp pack only when the pack's config has a trunk_quant block, as these packs' config.json does.
The plugin needs an M5-class GPU (NAX), and it refuses any other GPU by name at load. MTP needs
ddalcu/mlx-serve#1017, which is in 26.11.1 and later and in
7b13376d.
The plugin's plugin.json asks for mlx-serve 26.11.1 or later. But the plugins repo builds every plugin against a
plugin kit for mlx-serve 26.11.3 (its KIT_VERSION file), and on 2026-11-14 the latest release is 26.11.2. So the
repo's installer cannot download that kit yet, and the build steps in the plugin's
README stop there. Until
26.11.3 is released, build mlx-serve from source at commit 7b13376d and build its kit yourself.
Install the plugin from CommunityServe/plugins#52, which is open on
2026-11-14. #52 moves the repo's mlx-serve commit (MLX_SERVE_REF) from bc6c5f2c to 7b13376d. At build time, this plugin
copies part of the mlx-serve code at that commit. #52 adds 22 names to the plugin's cut list (scripts/cut.txt) and
updates the hash of that copy, so that the copy made at 7b13376d compiles. #52's CI compiles every plugin of the
repo against a 7b13376d kit. Main's bc6c5f2c is older than ddalcu/mlx-serve#1131, so the repo's CI cannot build
turboexl3_ds41 there. Our converter repository gives the full route in
Before mlx-serve 26.11.3 is released. In short:
# git clone --recurse-submodules https://github.com/ddalcu/mlx-serve && cd mlx-serve
# git checkout 7b13376d && git submodule update --init
# then build the server ("Prerequisites" and "Server only" in docs/building.md)
./.zig-toolchain/zig build kit -Dversion=26.11.3
export PATH="$PWD/zig-out/bin:$PATH"
# in the same shell, in a checkout of CommunityServe/plugins at pull request 52
# (git fetch origin pull/52/head && git checkout FETCH_HEAD)
rm -rf .kit
MLX_SERVE_KIT=/path/to/mlx-serve/zig-out/kit/mlx-serve-plugin-kit-26.11.3 scripts/install.sh turboexl3_qwen38
install.sh runs mlx-serve plugin install with the mlx-serve that is first on your PATH, so put your build
there first. That command puts the plugin in ~/.mlx-serve/plugins/. The
kit must come from the mlx-serve build that serves, because the plugin links no MLX of its own. rm -rf .kit clears a kit that an earlier install staged. The
server's --version can say 26.11.1; nothing compares it with the plugin's requires_mlx_serve. When 26.11.3 is
released with its kit and #52 (or a later change of MLX_SERVE_REF) is merged, install that server, then use main:
run rm -rf .kit and scripts/install.sh turboexl3_qwen38 without MLX_SERVE_KIT.
Then set "enabled": true in ~/.mlx-serve/plugins.json. After you download a branch and add its n-gram table (see
below), start the server from the same shell:
mlx-serve --model <pack folder> --serve
The plugin reads the shards past the OS file cache by itself when the pack is larger than half of the Mac's memory.
A log line that starts [plugin] turboexl3_qwen38 serves shows that the plugin took the pack. We have not served
through this exact route; #52's CI shows only that the plugins compile against that kit. The plugin's README gives
speeds that it measured on mlx-serve 7d52c147.
The repo README and
docs/using-plugins.md cover installing a
plugin and turning it on.
qwen38-exl3-full and the turboexl3 quant plugin
The packs were first measured on this route, which takes two parts.
The first part is mlx-serve with full-EXL3 support for Qwen3.8, which means the trunk_quant block, the EXL3 n-gram
table and the EXL3 MTP head. That support is the branch
qwen38-exl3-full of davidtai/mlx-serve, the head of
ddalcu/mlx-serve#997 before that pull request was cut down to a plugin
host change and merged.
The second part is the turboexl3 quant plugin. It is required, because it claims the pack's expert_quant and
trunk_quant blocks. Build it from the plugin/ directory
of davidtai/turboexl3.
Download a branch
Pass the branch name to --revision. This command downloads 5.05bpw, and the steps below continue with it:
hf download OpensourceWTF/Qwen3.8-Flash-Next-exl3 --revision 5.05bpw --local-dir ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve
Each branch's card gives the same steps with its own folder name.
The n-gram table
The branches do not include ngram_embedding.safetensors, the n-gram table, and each pack needs it in its folder.
mlx-serve maps the file and reads the rows each token needs, so the table costs disk space and OS file cache, not
resident memory.
For 4.05bpw and 5.05bpw, the table is turboderp's file, unchanged. Download ngram_embedding.safetensors from
turboderp/Qwen3.8-Flash-Next-exl3 at branch
4.05bpw_h6_ng6 or
5.05bpw_h6_ng6 and put it in the pack
folder as it is. Both branches ship the same file, with sha256
c63613666649014e668cdbd32a4e3f1e8beda697eb4f3a1874300d93a1bfbc75 and a size of 39,040,193,720 B. For the 5.05bpw
folder above, these commands fetch the file and check it:
hf download turboderp/Qwen3.8-Flash-Next-exl3 ngram_embedding.safetensors --revision 5.05bpw_h6_ng6 \
--local-dir ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve
shasum -a 256 ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve/ngram_embedding.safetensors
# expected: c63613666649014e668cdbd32a4e3f1e8beda697eb4f3a1874300d93a1bfbc75 (39,040,193,720 B)
For 2.05bpw and 3.05bpw, turboderp stores the table as 128 row shards in his ngram_embedding.safetensors, on
branches 2.05bpw_h4_ng4 and
3.05bpw_h5_ng5. mlx-serve reads one
tensor, so the converter makes the pack's table from his file. The finalize phase of its
scripts/restack_exl3_q2_full.py clones the file with APFS clonefile, so no data is copied, and rewrites only its
header to name the same bytes as one tensor. That phase also writes the carry shard, config.json and the index, which
the branch already has.
Never pass the downloaded pack folder to the converter as --hybrid (its base folder) or --out (its output
folder). Before the finalize phase links each file from --hybrid, it deletes the file of that name in --out. When
the two are the same folder, this deletes the pack's own files. The phase also rewrites config.json and the index in
--out. The converter's
docs/NGRAM-TABLE.md
gives a safe route, with the commands. It makes the table in a separate work folder, checks its sha256 and moves only
the table into the pack. We have not run that route. For 2.05 and 3.05, do not download turboderp's table into the
pack folder. His file has the same name but holds 128 row shards, which mlx-serve cannot read. If a file named
ngram_embedding.safetensors is already in the pack, delete it before you run the route. The route's mv -n does not
replace that file, and the route then deletes the new table. After the route, check the pack's table with the
route's hash_evict.py command and the full sha256 that NGRAM-TABLE.md gives.
Our results are a 3.05 table of 32,640,183,408 B (sha256 f7fa0ddfe932…) and a 2.05 table of 26,240,152,672 B (sha256
1f5d5d099b46…). NGRAM-TABLE.md gives the full sha256 of each.
Build the turboexl3 quant plugin and start the server
For the second way, build mlx-serve with the support listed in Requirements and its plugin kit, then
build the turboexl3 plugin from plugin/:
# in the mlx-serve checkout
zig build -Doptimize=ReleaseFast
zig build kit # stages zig-out/kit/mlx-serve-plugin-kit-<version>/
# in a checkout of github.com/davidtai/turboexl3
cd plugin
ln -s <mlx-serve checkout>/zig-out/kit/mlx-serve-plugin-kit-<version> kit
zig build # writes zig-out/lib/libturboexl3.dylib
mlx-serve plugin check zig-out/lib/libturboexl3.dylib --model ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve
# expected: claims expert_quant (by_marker), trunk_quant (by_marker)
mlx-serve plugin install zig-out/lib/libturboexl3.dylib
# then set "enabled": true in ~/.mlx-serve/plugins.json
Then start the server:
MLX_SERVE_NOCACHE_LOAD=1 mlx-serve --model ~/models/Qwen3.8-Flash-Next-EXL3-5.05-mlx-serve --serve
MLX_SERVE_NOCACHE_LOAD=1 reads the shards past the OS file cache, so the load does not also fill memory with cached
copies of the shards. Our measurements used it.
The default numerics use turboexl3's kernels, which reorder some sums. MLX_SERVE_BIT_EXACT=1 selects exact numerics:
the routed experts run on Sushi, the EXL3 kernels built into mlx-serve, and no sum is reordered. The two modes give
different text.
turboexl3 runs the routed experts only on an Apple M5 or later with macOS 26.2 or later. On other Macs, the plugin leaves the routed experts to Sushi, but we did not test this.
The plugin links no MLX of its own, so the MLX version of the kit must be the same as the server's.
Quality
| Branch | KLD, turboderp's | PPL, turboderp's | HumanEval pass@1 (of 164) | HumanEval+ (of 164) | MBPP pass@1 (of 378) | MBPP+ (of 378) |
|---|---|---|---|---|---|---|
5.05bpw |
0.0040 | 1.3848 | 0.988 (162) | 0.945 (155) | 0.955 (361) | 0.812 (307) |
4.05bpw |
0.0066 | 1.3897 | 0.988 (162) | 0.963 (158) | 0.958 (362) | 0.796 (301) |
3.05bpw |
0.0174 | 1.4074 | 0.982 (161) | 0.939 (154) | 0.950 (359) | 0.820 (310) |
2.05bpw |
0.0681 | 1.4871 | 0.970 (159) | 0.927 (152) | 0.958 (362) | 0.807 (305) |
The KLD and PPL columns are turboderp's. They are the mean KL divergence D_KL(p_BF16 ‖ p_quant) and the perplexity over a self-generated in-domain trace of 14,496 input and 45,588 output tokens, measured by turboderp with exllamav3 on his files. On that trace BF16 has a perplexity of 1.38474, and the KLD noise floor is 0.00262. The packs hold his quantized bytes, but these numbers were not measured through mlx-serve.
The EvalPlus columns are our end-to-end check through mlx-serve: EvalPlus on each pack, greedy pass@1, with the number of problems passed in parentheses. The setup is in the coding evals of turboexl3's results.
The charts below are turboderp's, linked from his model card. In them, the EXL3 series is his plain quants, and our packs are its 2.05, 3.05, 4.05 and 5.05 points. EXL3-SC is his self-calibrated quants, and the other series are other formats he measured.
Speed
The speed tables were measured on one Apple M5 Max with 128 GB under macOS 27.0.1. The server was our mlx-serve tree
with turboexl3, before its port to the current mlx-serve, on two builds. 5.05bpw ran on build S21, which is build
PR17B plus the later decode optimizations that the branch
qwen38-exl3-full ports, at MTP depth 4. 4.05bpw,
3.05bpw and 2.05bpw ran on build PR17B at MTP depth 3, and they were not re-measured on build S21. Compare branches
within one build only. These tables come from the second way that Requirements describes, not from the
turboexl3_qwen38 plugin.
How the speed was measured
Each cell is one run, with no repeats. For each prompt length and mode, a warm-up server first ran the prompt for one token, which puts the prompt's n-gram rows into the OS file cache. Then the other pack files were evicted from the cache, a new server started, the SoC cooled below 40 °C, and one request ran.
The request was the prompt as one user message in Qwen's chat template with thinking on, greedy (temperature 0), with seed 1234 and at most 1,024 new tokens. The prompts are fixed texts, Python source code with one coding task, and they are the same bytes for every pack and mode.
The server flags were --ctx-size 262144 --max-concurrent 1 --mtp-max-depth 3 --mtp-typical 0.2 --prefix-cache-mem 1GB,
with --mtp-max-depth 4 on build S21. The environment variables were MLX_SERVE_GPU_CEILING_MB=114688,
MLX_SERVE_CACHE_LIMIT=2147483648, MLX_SERVE_NOCACHE_LOAD=1 and MLX_SERVE_NGRAM_WARM=0.
The prefill and decode rates come from the server's log line for the request. TTFT is the client's time from sending the request to the first streamed token. Peak memory is the server's peak phys_footprint, which includes Metal memory but not the file cache of the n-gram table.
turboexl3 reorders sums, so the two modes give different answers. With MTP, their decode rates also depend on how many drafts each answer accepts.
In the tables, "turboexl3" is the default numerics and "exact" is MLX_SERVE_BIT_EXACT=1. Prefill and decode are in
tokens per second, and * marks a run whose answer reached the 1,024-token cap.
Build PR17B, MTP depth 3: 4.05bpw, 3.05bpw and 2.05bpw
None of the 18 runs of 4.05bpw reached the cap, against one of 18 for 3.05bpw and one of 18 for 2.05bpw.
Prefill, tok/s
| prompt | tokens | 4.05bpw, turboexl3 | 4.05bpw, exact | 3.05bpw, turboexl3 | 3.05bpw, exact | 2.05bpw, turboexl3 | 2.05bpw, exact |
|---|---|---|---|---|---|---|---|
| 1K | 1,031 | 1373.0 | 1025.5 | 1572.3 | 1580.5 | 1603.2 | 1648.9 |
| 2K | 2,013 | 1423.7 | 1916.3 | 2080.1 | 2057.0 | 2114.1 | 2147.0 |
| 4K | 4,025 | 2116.1 | 1938.8 | 2497.5 | 2464.7 | 2554.9 | 2518.1 |
| 8K | 8,100 | 2711.0 | 2488.6 | 2769.8 | 2498.3 | 2809.4 | 2412.1 |
| 16K | 16,275 | 2830.0 | 2538.2 | 2834.5 | 2599.3 | 2771.8 | 2714.5 |
| 32K | 32,424 | 2759.3 | 2492.8 | 2766.3 | 2578.8 | 2767.0 | 2664.4 |
| 64K | 64,987 | 2639.1 | 2485.6 | 2659.6 | 2484.4 | 2680.0 | 2543.3 |
| 128K | 129,949 | 2512.4 | 2398.9 | 2522.2 | 2399.0 | 2544.4 | 2451.4 |
| 256K | 258,992 | 2344.0 | 2270.0 | 2348.0 | 2273.7 | 2362.3 | 2318.8 |
Decode, tok/s
| prompt | tokens | 4.05bpw, turboexl3 | 4.05bpw, exact | 3.05bpw, turboexl3 | 3.05bpw, exact | 2.05bpw, turboexl3 | 2.05bpw, exact |
|---|---|---|---|---|---|---|---|
| 1K | 1,031 | 141.0 | 119.7 | 140.0 | 131.8 | 131.5 | 126.2 |
| 2K | 2,013 | 134.8 | 114.2 | 131.3* | 126.6 | 123.6* | 119.6 |
| 4K | 4,025 | 132.3 | 116.5 | 135.2 | 124.5 | 125.3 | 113.3 |
| 8K | 8,100 | 135.7 | 116.1 | 134.4 | 128.2 | 127.6 | 118.2 |
| 16K | 16,275 | 137.6 | 112.5 | 132.3 | 126.6 | 120.6 | 110.7 |
| 32K | 32,424 | 126.4 | 108.7 | 128.5 | 117.8 | 117.8 | 107.5 |
| 64K | 64,987 | 119.5 | 102.5 | 116.6 | 112.0 | 110.9 | 108.5 |
| 128K | 129,949 | 114.3 | 96.8 | 110.4 | 106.3 | 101.3 | 98.9 |
| 256K | 258,992 | 105.5 | 89.6 | 99.4 | 93.6 | 96.0 | 91.8 |
TTFT, s
| prompt | tokens | 4.05bpw, turboexl3 | 4.05bpw, exact | 3.05bpw, turboexl3 | 3.05bpw, exact | 2.05bpw, turboexl3 | 2.05bpw, exact |
|---|---|---|---|---|---|---|---|
| 1K | 1,031 | 0.76 | 1.01 | 0.66 | 0.66 | 0.65 | 0.63 |
| 2K | 2,013 | 1.42 | 1.06 | 0.98 | 0.99 | 0.96 | 0.95 |
| 4K | 4,025 | 1.91 | 2.09 | 1.62 | 1.64 | 1.59 | 1.61 |
| 8K | 8,100 | 3.00 | 3.27 | 2.94 | 3.26 | 2.90 | 3.37 |
| 16K | 16,275 | 5.77 | 6.43 | 5.76 | 6.28 | 5.89 | 6.01 |
| 32K | 32,424 | 11.77 | 13.03 | 11.74 | 12.60 | 11.74 | 12.19 |
| 64K | 64,987 | 24.66 | 26.18 | 24.47 | 26.19 | 24.28 | 25.59 |
| 128K | 129,949 | 51.78 | 54.23 | 51.58 | 54.23 | 51.13 | 53.07 |
| 256K | 258,992 | 110.60 | 114.20 | 110.41 | 114.02 | 109.74 | 111.80 |
Peak memory, GiB, turboexl3
| prompt | tokens | 4.05bpw | 3.05bpw | 2.05bpw |
|---|---|---|---|---|
| 1K | 1,031 | 71.5 | 56.7 | 41.9 |
| 2K | 2,013 | 71.1 | 56.4 | 41.6 |
| 4K | 4,025 | 71.3 | 56.7 | 42.9 |
| 8K | 8,100 | 73.5 | 59.1 | 42.4 |
| 16K | 16,275 | 75.1 | 60.2 | 45.4 |
| 32K | 32,424 | 75.7 | 60.8 | 46.1 |
| 64K | 64,987 | 77.2 | 62.3 | 47.8 |
| 128K | 129,949 | 79.9 | 65.4 | 50.8 |
| 256K | 258,992 | 84.8 | 70.4 | 55.6 |
Build S21, MTP depth 4: 5.05bpw
Three of the 18 runs reached the cap. At 256K the server narrowed its prefill chunk to 4,096 tokens, against 8,192 at the other lengths.
| prompt | tokens | prefill, turboexl3 | prefill, exact | decode, turboexl3 | decode, exact | TTFT s, turboexl3 | TTFT s, exact | peak GiB, turboexl3 |
|---|---|---|---|---|---|---|---|---|
| 1K | 1,031 | 1093.7 | 955.7 | 134.9 | 126.8 | 0.95 | 1.09 | 86.2 |
| 2K | 2,013 | 1625.5 | 1161.2 | 126.5* | 122.0* | 1.25 | 1.74 | 85.9 |
| 4K | 4,025 | 1987.2 | 1884.1 | 138.2 | 123.6 | 2.04 | 2.15 | 86.5 |
| 8K | 8,100 | 2555.2 | 2018.0 | 126.6 | 123.1 | 3.18 | 4.03 | 87.9 |
| 16K | 16,275 | 2711.8 | 2264.2 | 124.2* | 122.3 | 6.02 | 7.20 | 90.1 |
| 32K | 32,424 | 2674.0 | 2334.2 | 122.8 | 114.3 | 12.15 | 13.91 | 90.2 |
| 64K | 64,987 | 2550.7 | 2242.2 | 113.6 | 107.8 | 25.51 | 29.02 | 92.2 |
| 128K | 129,949 | 2500.4 | 2179.0 | 108.9 | 103.6 | 52.03 | 59.70 | 94.9 |
| 256K | 258,992 | 2329.1 | 2073.1 | 95.2 | 92.3 | 111.31 | 125.03 | 100.1 |
Serial decode
The serial-decode runs used --no-mtp in place of the two MTP flags and the same method otherwise. Four of the 18 runs
reached the cap.
| prompt | tokens | prefill, turboexl3 | prefill, exact | decode, turboexl3 | decode, exact | TTFT s, turboexl3 | TTFT s, exact | peak GiB, turboexl3 |
|---|---|---|---|---|---|---|---|---|
| 1K | 1,031 | 1157.9 | 1014.1 | 55.9 | 56.8 | 0.90 | 1.03 | 84.0 |
| 2K | 2,013 | 1578.1 | 1439.9 | 55.0* | 56.0* | 1.29 | 1.41 | 83.3 |
| 4K | 4,025 | 2173.5 | 1756.9 | 55.1* | 56.2* | 1.86 | 2.30 | 83.3 |
| 8K | 8,100 | 2609.6 | 2211.2 | 54.9 | 55.9 | 3.12 | 3.68 | 85.5 |
| 16K | 16,275 | 2751.4 | 2277.3 | 54.9 | 55.8 | 5.93 | 7.16 | 87.2 |
| 32K | 32,424 | 2698.0 | 2290.3 | 54.2 | 55.1 | 12.04 | 14.18 | 87.7 |
| 64K | 64,987 | 2604.9 | 2269.8 | 53.2 | 54.2 | 24.98 | 28.67 | 88.9 |
| 128K | 129,949 | 2503.1 | 2203.3 | 52.0 | 53.1 | 51.97 | 59.04 | 91.1 |
| 256K | 258,992 | 2345.8 | 2094.2 | 51.2 | 51.9 | 110.51 | 123.78 | 95.8 |
How the packs were made
In every branch, the quantized weights are turboderp's bytes, rearranged into mlx-serve's file layout. The n-gram table
is his too, but it is not in the branches, as The n-gram table explains. The unquantized f16
tensors were rounded and are stored as bf16, except the n-gram head_bias, which stays f16.
Three files are ours: the carry shard (trunk-exl3-carry.safetensors), config.json and
model.safetensors.index.json. The vision shard (trunk-model-vision.safetensors) and the small files come from our
earlier pack.
The conversion scripts, in the qwen38/ folder of OpenSourceWTF/mlx-serve-streaming-pack-converters, do the
repack. That folder's docs/PROVENANCE.md names the script revisions that built each branch, with their sha256. Each
branch's card lists every file with its source and the change made to it.
Credits
The model, the tokenizer, the chat template and the license are Qwen's. turboderp made the EXL3 quantizations that these packs are made from, with exllamav3. EXL3 is based on QTIP by Cornell RelaxML.
ddalcu wrote mlx-serve and its Qwen3.8 model code, and made Sushi, the EXL3 kernels built into mlx-serve. The pack ddalcu/Qwen3.8-Flash-Next-MLX-Serve-mixed-4-8bit is the source of the packs' vision tower and base config.
turboexl3 provides the EXL3 Metal kernels of the default mode. The conversion
scripts are in the qwen38/ folder of
OpenSourceWTF/mlx-serve-streaming-pack-converters.
License
The weights are derived from Qwen3.8-Flash-Next and stay under the Qwen Community License 1.0. That license is the
LICENSE file, the same as in Qwen's repository, and turboderp's and ddalcu's repositories use the same license.
The LICENSE file is the license, and what follows is only a short summary of it. Keep the copyright notice and the
permission notice with every copy. A commercial product or service that uses the model and has more than 100,000,000
monthly active users, or more than US$20,000,000 monthly revenue, must show the model name in its user interface.
A licensee that runs, or whose affiliate runs, a "Model as a Service" or "AI Work Assistant" business (as the license defines them) needs a separate license from Qwen before any commercial use. Internal use that gives third parties no access to the model, its outputs or its capabilities does not need one. There is no warranty.
The code (mlx-serve, Sushi, turboexl3, exllamav3 and the conversion scripts) has its own licenses.
Model tree for OpensourceWTF/Qwen3.8-Flash-Next-exl3
Base model
Qwen/Qwen3.8-Flash-Next

