OpensourceWTF/Ling-3.1-flash-streaming-repack-exl3

Ling-3.1-flash, streaming EXL3 packs for mlx-serve

Each branch of this repo holds one of turboderp's EXL3 quantizations of inclusionAI's Ling-3.1-flash. turboderp made the quantizations with exllamav3 and publishes them in turboderp/Ling-3.1-flash-exl3. We repacked each one so that mlx-serve can read the routed experts from the SSD when it needs them. The quantized weights are his, byte for byte. We did not quantize anything again, so please credit turboderp when you use them. Each bitrate is a branch. main holds only this card, the license and Hugging Face's .gitattributes.

Ling-3.1-flash is a mixture-of-experts (MoE) model. inclusionAI's card gives it 560B total parameters and 25B active parameters per token. Each pack (153.5 or 188.3 GB) is larger than the memory of a 128 GB Mac. mlx-serve keeps the rest of the model and part of the routed experts in GPU memory, and it reads the other experts from the SSD. A quant plugin, turboexl3_ling, gives mlx-serve the EXL3 format and its Metal kernels. It needs an M5-class GPU.

We are OpensourceWTF (OpenSourceWTF on GitHub). We last updated this card on 2026-11-14. Terms explains the words that this card uses.

State of each part

The plugin that serves these packs is in an open pull request, CommunityServe/plugins#53. You can install it from that pull request's branch, as Build mlx-serve and install the plugin shows. This table gives the state of each part.

Part	State
The packs	On this repo, one branch for each bitrate
mlx-serve's plugin interface, ddalcu/mlx-serve#1192	Merged into main on 2026-11-13. No release has it. The latest release, v26.11.2, is from 2026-11-06.
ddalcu/mlx-serve#1195: runs the attention of the MTP draft check in one GPU kernel launch instead of several	Open, ready for review
ddalcu/mlx-serve#1205: runs the prompt one layer at a time, which gives about 3 times the prefill speed	Open, ready for review. It builds on #1195.
The turboexl3_ling 0.2.0 quant plugin, CommunityServe/plugins#53	Open, ready for review. It builds on #52.
CommunityServe/plugins#52, which builds that repo's plugin kit from mlx-serve 7b13376d	Open. #53 needs it.
The program that makes a bank-fit.json	Not public (The bank's memory)

What speed to expect

These are measurements of 2.60bpw on our Apple M5 Max with 128 GB. Our standard test is one coding prompt of 18,380 tokens and a greedy reply of up to 1,024 tokens. The memory plan is either mlx-serve's bills, which is what you get, or a fit that we made on our Mac (The bank's memory). Speed has every run and the method.

mlx-serve build	Memory plan	Runs	Prefill tok/s	Decode tok/s	Slot rows
48c886ee, the head of #1205	the bills	1 request	339.70	11.36	149 prompt / 157 decode
48c886ee, the head of #1205	our fit	standard test, mean of 4	341.85	11.89	183-184
cbb7f16c, #1192 and #1195 without #1205	our fit	standard test, mean of 4	112.9	11.96	185-186

On 48c886ee, the request without a fit ran close to the fitted standard test. That is one request against a mean of four runs, not a controlled comparison. We did not run the standard test on main 7b13376d alone. Without #1205, mlx-serve runs the prompt in chunks, as cbb7f16c does, so expect about 113 tok/s prefill. One request without a fit on 56460793, a build of #1192 without #1205, decoded at 10.25 tok/s.

Branches

Branch	Routed experts	Trunk	Size	Source and credit	Served by	Status
2.60bpw	EXL3 2.5 bpw (K2.5), layers 4-71	K6; K4 for the dense MLPs of layers 0-3	188.3 GB	turboderp's 2.60bpw branch (revision 5368b6b9)	turboexl3_ling 0.2.0 (CommunityServe/plugins#53)	live
2.10bpw	EXL3 2 bpw (K2), layers 4-71	K6; K4 for the dense MLPs of layers 0-3	153.5 GB	turboderp's 2.10bpw branch (revision ebce035f)	turboexl3_ling 0.2.0 (CommunityServe/plugins#53); not tested with this pack	live

Sizes are in GB, where 1 GB is 10^9 bytes. K is the number of bits per weight of an EXL3 tensor. A branch's name is turboderp's average over all weights: his config.json gives 2.59 and 2.1. The trunk is every weight that is not a routed expert of layers 4-71 or the MTP layer: attention, the shared experts, the dense layers and the output head. The MTP (multi-token prediction) layer drafts tokens for the main model to check.

All speed results on this card are for 2.60bpw. The plugin's claim rule (model type bailing_hybrid and an EXL3 quantization_config entry in config.json) also matches 2.10bpw, but we did not load 2.10bpw with turboexl3_ling 0.2.0. Each branch's card lists its files and how they were made.

Sources

Repo	What it holds
turboderp/Ling-3.1-flash-exl3	turboderp's EXL3 quantizations for exllamav3, a branch per bitrate, with his KL divergence chart. It is the source of every branch here.
inclusionAI/Ling-3.1-flash	The original model, its model card and its recommended sampling settings.
OpenSourceWTF/mlx-serve-streaming-pack-converters	The converters for several models. Its folder ling31/ holds scripts/ling_exl3_to_streaming.py, which made both packs. Commit 899fe84 is the committed version of the script. 2.10bpw ran a copy byte-identical to it, and 2.60bpw ran uncommitted copies (How we made the packs).

How to download and run

Requirements

• A Mac with an M5-class GPU. turboexl3's prompt kernels run on the matrix units of the M5 GPU (NAX). On another GPU, the plugin refuses to load with an M5-class GPU (NAX) is needed. We ran macOS 26.6.
• Memory. We measured only one Apple M5 Max with 128 GB. We do not know if a Mac with less memory can serve the packs.
• Disk. 188.3 GB for 2.60bpw, or 153.5 GB for 2.10bpw.
• mlx-serve, built from source. Use main at 7b13376d (the commit that the plugin's pull request builds against), or the head of #1205, 48c886ee, for the faster prefill. 48c886ee holds main 7b13376d, #1195 and #1205.
• The turboexl3_ling 0.2.0 plugin, from the branch of CommunityServe/plugins#53.

The packs do not load in exllamav3, transformers, vLLM, SGLang or mlx-lm. For exllamav3, use turboderp's repository.

Download a branch

Install the Hugging Face CLI first (pip install -U huggingface_hub). Pass the branch name to --revision. This command downloads 2.60bpw:

hf download OpensourceWTF/Ling-3.1-flash-streaming-repack-exl3 --revision 2.60bpw --local-dir ~/models/Ling-3.1-flash-streaming-repack-exl3-2.60bpw

Build mlx-serve and install the plugin

These commands build mlx-serve from main and its plugin kit:

git clone --recurse-submodules https://github.com/ddalcu/mlx-serve && cd mlx-serve
git checkout 7b13376d && git submodule update --init
brew bundle install --file=Brewfile
./scripts/fetch-zig.sh && ./scripts/fetch-llama.sh && ./scripts/build-mlx.sh
.zig-toolchain/zig build -Doptimize=ReleaseFast && .zig-toolchain/zig build kit

For the faster build with #1195 and #1205, replace the second line with this one:

git fetch origin pull/1205/head && git checkout 48c886ee && git submodule update --init

The server is zig-out/bin/mlx-serve, and the plugin kit is in zig-out/kit/. In the same shell, point the plugin build at that kit, then install the plugin from the branch of #53:

KIT=$(ls -d $PWD/zig-out/kit/mlx-serve-plugin-kit-*/) && export MLX_SERVE_KIT=${KIT%/}
export PATH=$PWD/zig-out/bin:$PATH
cd ..
git clone https://github.com/CommunityServe/plugins && cd plugins
git fetch https://github.com/davidtai/plugins add-turboexl3_ling-quant-k7b && git checkout FETCH_HEAD
scripts/install.sh turboexl3_ling

scripts/install.sh builds the plugin, runs its unit tests and installs it. Then set "enabled": true in ~/.mlx-serve/plugins.json. The plugin's README in that branch gives more detail. At load, the log shows [plugins] loaded turboexl3_ling.

Start the server

These are the settings that we measured with, on a Mac with 128 GB. On a Mac of another size, leave out the sysctl line and MLX_SERVE_GPU_CEILING_MB. mlx-serve then plans to the Mac's own GPU limit. We did not test this. Run the command in the shell of the steps above, which put the mlx-serve that you built on the PATH.

sudo sysctl iogpu.wired_limit_mb=114688
MLX_SERVE_GPU_CEILING_MB=114688 MLX_SERVE_CACHE_LIMIT=2147483648 \
  mlx-serve --serve --model ~/models/Ling-3.1-flash-streaming-repack-exl3-2.60bpw \
  --ctx-size 32768 --prefill-chunk 4096 --mtp-max-depth 3

• iogpu.wired_limit_mb=114688 lets the GPU use up to 112 GiB until the next restart.
• MLX_SERVE_GPU_CEILING_MB is the GPU memory that mlx-serve plans to (the ceiling). The log shows the line [diag] GPU ceiling forced to 114688 MB, which is expected.
• MLX_SERVE_CACHE_LIMIT limits MLX's buffer cache to 2 GiB. On 2026-11-13, our standard test without it ran prefill at 115.95 tok/s against 112.88 tok/s with it. But without it, the Mac's free memory fell below 1 GB, and macOS paged memory out. All our other turboexl3_ling 0.2.0 results used the limit.
• --ctx-size is the longest prompt plus its reply. With 32,768, the prompt and the reply together can be up to 32,768 tokens.
• --prefill-chunk 4096 is an upper limit. mlx-serve keeps one attention layer's scores under 4 GiB, so at 18,380 prompt tokens it uses chunks of 1,536 tokens. With #1205, it runs the whole prompt one layer at a time.
• --mtp-max-depth 3 lets the MTP layer draft up to 3 tokens in each round.

The server listens on port 11234, with an OpenAI-style API. At load, the log shows [turboexl3_ling] streamed experts: <P> prompt / <D> decode slot rows per layer (<B> B a row). This command sends one request:

curl http://127.0.0.1:11234/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "max_tokens": 256}'

The bank's memory, and why there is no bank-fit.json

The plugin's bank holds routed experts in GPU memory, in slot rows. A slot row holds one expert of each of the 68 routed layers, without the records' padding: 336,427,008 B for 2.60bpw. The bank takes the memory that the rest of the server leaves. mlx-serve sets the bank's size in one of two ways:

• From its bills. A bill is mlx-serve's upper-bound estimate of the memory of the prompt pass or of decode. This is what you get with these packs.
• From a fit. A bank-fit.json in the pack's folder holds the measured memory peaks of one request of our standard test (18,380 + 1,024 tokens). In our runs, a fit gave 25 to 35 more slot rows than the bills.

We do not ship a bank-fit.json, because a fit from another build can plan a bank larger than the memory. mlx-serve main (7b13376d) applies a pack's fit when its format, its ceiling and its cell size match the server's (parseBankFit in src/sdk/quant/outside_model.zig). It does not check which mlx-serve build or plugin made the fit, and our fits differ by build. The program that makes a fit is not public yet, so you cannot make your own fit. Without a fit, the bank is smaller, but its plan comes from the bills.

The fits and the requests without a fit

A fit names its cell, mlx-serve's reference request of a 16,384-token prompt plus a 1,024-token reply, and the server's cell must match. The fit's bills grow from the tokens of the request that it measured. The base of a phase is its measured memory peak without the bank. On the same Mac at the same ceiling, our fits gave these decode bases:

mlx-serve build	Decode base GB
56460793, a commit of #1192	42.71
48c886ee, the head of #1205	44.87

On the builds of this card, we measured without a fit only these requests. Each was one greedy request of our standard test's prompt, with a memory sampler running. Where we ran the fitted standard test of the same build, it is in the next row. A fitted run held the same number of slot rows in both phases. Everything else without a fit is not measured.

mlx-serve build	Fit	Runs	Prefill tok/s	Decode tok/s	Slot rows, prompt / decode
56460793 (#1192)	none	1 request	112.52	10.25	162 / 162
56460793 (#1192)	ours	standard test, mean of 4	113.1	11.28	191-192
3f8c8a57 (#1192, #1195, #1205)	none	1 request	338.22	11.19	151 / 159
3f8c8a57 (#1192, #1195, #1205)	ours	standard test, mean of 4	339.7	11.64	184-185
48c886ee (#1205's head)	none	1 request	339.70	11.36	149 / 157
48c886ee (#1205's head)	ours	standard test, mean of 4	341.85	11.89	183-184

On 3f8c8a57 and on 48c886ee, the request without a fit ran close to the fitted standard test, with 25 to 35 fewer slot rows. Each is one request against a mean of four runs, not a controlled comparison. On 56460793, the request without a fit decoded at 10.25 tok/s, against 11.28 tok/s with the fit.

Quality

The packs hold turboderp's quantized bytes, so his KL divergence chart describes their weights in exllamav3. It is on his model card. We did not measure quality on mlx-serve's own Ling model with the turboexl3_ling 0.2.0 plugin.

An earlier test, on 2026-11-08, used the old whole-model plugin. That plugin had its own copy of the Ling model, and its pull request, CommunityServe/plugins#50, is closed. The old plugin is not available, and we do not support it. The test served 2.60bpw and 2.10bpw on a build of it from before its version 0.1.0. It compared the plugin's next-token distributions on one 1,146-token prompt with exllamav3's, for turboderp's 2.60 bpw quant on another machine.

Run against exllamav3's 2.60 bpw logits	Median KL	Top-1 agreement	Top-20 overlap
A second exllamav3 run (the reference's own spread)	0.00286	90.74%	88.41%
2.60bpw on the old plugin	0.00433	89.52%	86.63%
2.10bpw on the old plugin	0.05926	79.18%	68.60%

Top-1 agreement is the share of positions where the run's most likely token is the reference's. Top-20 overlap is the share of the reference's 20 most likely tokens that are also in the run's 20. KL is in nats, over the run's 256 most likely tokens at each position plus one bucket for all other tokens. The second exllamav3 run put a different number of MoE layers on the CPU, which changes the rounding. Its row shows how much rounding alone moves these values.

The reference is the 2.60 bpw quant, so the 2.10bpw row shows that its output differs more from that quant's. It is not a quality score for 2.10bpw.

Speed

All results in this section are for 2.60bpw on one Apple M5 Max with 128 GB under macOS 26.6, with plugin turboexl3_ling 0.2.0 (source commit 79baca4; CommunityServe/plugins#53 holds the same code). TTFT is the time to the first token.

The standard test on 48c886ee

On 2026-11-14 we ran our standard test on 48c886ee, the head of #1205, with a fit that we made on that build. It was arm A of an A/B test whose other arm was a change that we did not keep.

Build, mean of 4 runs	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB	Slot rows
48c886ee	341.85	11.89	53.78	140.1	59.3%	95.28	183-184

Run	Seed	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB
1	1234	343.62	11.83	53.50	140.26	533 / 949 = 56.2%	95.2
4	4321	342.65	11.92	53.65	139.77	533 / 945 = 56.4%	95.2
6	1234	339.33	11.94	54.18	140.18	530 / 809 = 65.5%	95.5
7	4321	341.79	11.87	53.79	140.27	530 / 894 = 59.3%	95.2

All 4 runs gave the text with sha256 9632d6ec59247438…. The run numbers are the arm's positions in the A/B.

What #1205 changes

On 2026-11-14 we also ran an A/B test of #1205's changes. An A/B test compares two builds, arm A and arm B, on the same test.

• A is mlx-serve cbb7f16c: #1192 with #1195, before we merged main 7b13376d into them. It runs the prompt in chunks.
• B is mlx-serve 3f8c8a57: A plus the commits of #1205, which run the whole prompt one layer at a time.

48c886ee holds the same changes as B, with main 7b13376d merged in.

Arm, mean of 4 runs	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB	Slot rows
A, chunks (cbb7f16c)	112.9	11.96	162.8	248.7	55.5%	96.15	185-186
B, whole-prompt pass (3f8c8a57)	339.7	11.64	54.1	142.3	63.1%	95.65	184-185

Both arms gave the same text on both seeds. With #1205, prefill is 3.0 times as fast, and each request takes 106 s less. Decode is 2.7% slower. The first decode rounds after #1205's prompt pass are slower, and we do not know why yet. A held one slot row more than B.

SSD reads in the prompt, mean of 4 runs	A	B
TB read	1.514	0.197

B's 0.197 TB includes 35.9 GB of experts that B read early, on a prediction of the routes that the layer then did not use.

How the speed was measured

The standard test sends one prompt of 18,380 tokens: CPython's shlex.py and a coding task. The reply is up to 1,024 tokens, greedy (temperature 0). Each arm ran 4 times, in the order A B B A, then B A A B, on seeds 1234 and 4321. With greedy decoding, the seed does not change the text. Each run used a new server, and the chip cooled to 40 °C before the request. The server settings were those of Start the server.

Each arm had its own bank-fit.json, made on our Mac. We made the fits of 48c886ee and of B on their own builds. We made A's fit on 56460793, an earlier build of #1192. That is the kind of reuse across builds that the bank section warns about. A's peak footprint stayed at 96.15 GB, under the ceiling of 112 GiB (120.3 GB).

Prefill and decode are the server's own rates for the request. The per-run rates of the A/B tests above come from the timings of the server's response, as in #1205's description. The per-run rates of the earlier tests come from the server's log line, which gives one decimal, as in #1195's description. Means are rounded to 4 significant figures. The client measures TTFT and wall time. MTP acceptance is accepted draft tokens divided by drafted tokens; a mean is the mean of the runs' values. The drafted count changes from run to run, because MTP's planner chooses the draft length from round times that it measures. Peak footprint is the server process's peak memory, in GB.

Each run of the #1205 A/B

Run	Arm	Seed	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB
1	A	1234	112.78	11.76	162.99	250.27	531 / 955 = 55.6%	96.3
2	B	1234	341.83	11.62	53.78	142.13	536 / 951 = 56.4%	95.5
3	B	4321	340.87	11.87	53.93	140.42	525 / 800 = 65.6%	95.5
4	A	4321	112.85	12.10	162.89	247.72	531 / 960 = 55.3%	96.0
5	B	1234	342.04	11.63	53.75	142.00	524 / 753 = 69.6%	95.8
6	A	1234	112.97	12.01	162.71	248.18	532 / 957 = 55.6%	96.3
7	A	4321	112.96	11.96	162.73	248.57	534 / 962 = 55.5%	96.0
8	B	4321	334.07	11.46	55.03	144.59	527 / 868 = 60.7%	95.8

Earlier tests against the old whole-model plugin

On 2026-11-13 we ran the standard test twice, each time against the old whole-model plugin, turboexl3_ling 0.1.0 on mlx-serve 9a173773. That plugin had its own copy of the Ling model and a whole-prompt pass, and it ran with its own memory bill and no fit. The 0.2.0 arm ran plugin 79baca4 with our fit, the cache limit and the ceiling.

• Test 1: the 0.2.0 arm is mlx-serve 56460793, a commit of #1192.
• Test 2: the 0.2.0 arm is test 1's build plus #1195's kernels (37159073, a local build that is not public).

Test, arm, mean of 4 runs	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB
1, 0.1.0	701.9	25.95	26.20	65.82	60.7%	110.4
1, 0.2.0 (#1192)	113.1	11.28	162.5	253.7	66.2%	98.23
2, 0.1.0	701.5	25.85	26.22	65.93	60.6%	110.5
2, 0.2.0 (#1192 and #1195)	112.9	12.68	162.8	244.0	60.3%	98.00

In every run, the 0.1.0 arm decoded a text with sha256 09571f6a…, and the 0.2.0 arm a text with sha256 9632d6ec…. So the decode rates compare different texts. In decode, the 0.2.0 bank held 191-192 slot rows and the 0.1.0 bank held 236. The old plugin also had trunk and MTP optimizations that mlx-serve does not have. We did not measure the two causes apart.

Test 2's 0.2.0 arm decoded at 12.68 tok/s, and arm A of the A/B above, with the same changes, at 11.96 tok/s. The two builds differ in their base commit, their fit and their slot rows. We have not analyzed the difference.

Test 1, each run:

Run	Arm	Seed	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB
1	0.1.0	1234	701.0	26.00	26.23	65.75	448 / 734 = 61.0%	110.3
2	0.2.0	1234	113.3	11.40	162.23	252.59	519 / 757 = 68.6%	98.0
3	0.2.0	4321	113.1	11.20	162.53	254.56	502 / 720 = 69.7%	98.3
4	0.1.0	4321	701.5	26.00	26.21	65.77	450 / 741 = 60.7%	110.5
5	0.2.0	1234	113.0	11.10	162.65	255.35	510 / 844 = 60.4%	98.3
6	0.1.0	1234	700.5	26.00	26.25	65.82	448 / 732 = 61.2%	110.4
7	0.1.0	4321	704.7	25.80	26.10	65.94	444 / 742 = 59.8%	110.5
8	0.2.0	4321	113.1	11.40	162.46	252.37	517 / 785 = 65.9%	98.3

Test 2, each run:

Run	Arm	Seed	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB
1	0.1.0	1234	701.0	26.00	26.23	65.73	467 / 754 = 61.9%	110.5
2	0.2.0	1234	112.6	12.70	163.18	244.09	531 / 939 = 56.5%	98.0
3	0.2.0	4321	113.0	12.40	162.63	245.71	520 / 800 = 65.0%	98.0
4	0.1.0	4321	701.8	26.00	26.21	65.70	451 / 736 = 61.3%	110.4
5	0.2.0	1234	113.4	12.80	162.15	242.28	533 / 962 = 55.4%	98.0
6	0.1.0	1234	700.2	25.70	26.26	66.22	444 / 746 = 59.5%	110.5
7	0.1.0	4321	702.9	25.70	26.16	66.06	444 / 742 = 59.8%	110.4
8	0.2.0	4321	112.5	12.80	163.40	243.88	531 / 824 = 64.4%	98.0

Context sweep, 1K to 32K, on the chunk build

From 2026-11-13 to 2026-11-14 we ran a context sweep with plugin 79baca4 on mlx-serve cbb7f16c, arm A of the A/B above. That build runs the prompt in chunks, without #1205, so its prefill is slower than B's. The server had the standard test's settings, with --ctx-size at the prompt plus 16,384 tokens, and max_tokens was 16,384. Each size has its own coding prompt. Each size ran 4 times greedy and 4 times at inclusionAI's recommended sampling settings (temperature 1.0, top-p 0.95, top-k 20), on seeds 1234 and 4321. Every run ended on a stop token. The sweep stops at 32K. We will run 64K and 128K on the build that we publish with the plugin.

Greedy, mean of 4 runs. The 4 runs of each size gave one text.

Size	Prompt tokens	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB	Reply tokens
1K	1,058	77.50	11.85	13.66	507.4	61.5%	99.95	5,854
2K	2,093	128.1	11.53	16.35	319.7	52.8%	99.47	3,489
4K	4,181	199.9	11.50	20.92	261.3	55.9%	98.70	2,758
8K	8,397	132.6	11.30	63.34	536.6	58.6%	98.90	5,348
16K	16,800	102.6	11.40	163.8	886.9	60.5%	96.50	8,240
32K	33,591	54.75	6.550	613.6	2,394	60.8%	103.4	11,198

At inclusionAI's sampling settings, mean of 4 runs. Each run gave its own text, so the reply length is a mean.

Size	Prompt tokens	Prefill tok/s	Decode tok/s	TTFT s	Wall time s	MTP acceptance	Peak footprint GB	Reply tokens
1K	1,058	77.50	11.70	13.66	502.3	56.8%	99.85	5,685
2K	2,093	127.7	11.83	16.39	512.3	57.0%	100.2	5,844
4K	4,181	199.6	11.90	20.95	366.6	54.6%	99.42	4,132
8K	8,397	132.7	11.73	63.31	503.1	60.8%	99.05	5,121
16K	16,800	102.7	11.10	163.6	505.6	57.7%	94.10	3,801
32K	33,591	54.62	7.575	615.0	1,293	63.5%	86.85	4,204

We have not analyzed four results yet:

• At 32K, decode fell to 5.2 to 8.8 tok/s in the greedy runs and 5.1 to 9.8 tok/s in the sampled runs.
• Prefill fell from 199.9 tok/s at 4K to 132.6 tok/s at 8K.
• At 32K, the sampled runs' peak footprint was 86.85 GB, against 103.4 GB for greedy.
• At 16K, prefill was 102.6 tok/s, against 112.9 tok/s for the standard test's longer prompt on the same build.

The macOS memory pressure level stayed normal in every run.

Limits

• Prefill without #1205 is slow. Without #1205, each prompt chunk reads most experts from the SSD again. #1205 fixes this, but it is an open pull request, not part of main.
• Decode and prefill are slower than the old plugin's. The old whole-model plugin decoded at about 25.9 tok/s, against 11.89 tok/s on 48c886ee. It held more slot rows, and it had trunk and MTP optimizations that mlx-serve does not have yet. Its prefill was about 702 tok/s, against 341.85 tok/s. We have not analyzed the prefill difference.
• A long reply can pass the memory plan. When a reply grows past the KV cache that mlx-serve reserved, mlx-serve grows the cache. The old and new buffers then exist together for a moment. By calculation from the code, this can take memory about 2.7 GB past the plan at --ctx-size 32768. We did not measure it.
• Context. The packs' config.json sets a window of 262,144 tokens. The prompt and the reply must fit in it together. We measured prompts only up to 33,591 tokens.

These are not measured:

• the standard test on main 7b13376d alone;
• the full standard test (4 runs per arm) without a fit: we ran single requests only;
• the standard test at inclusionAI's sampling settings, and a context sweep, on a build with #1205;
• 2.10bpw with turboexl3_ling 0.2.0;
• more than one request at a time, and image input;
• any Mac but ours.

How we made the packs

The converter's scripts/ling_exl3_to_streaming.py made both packs on 2026-11-06. It runs on the CPU and copies bytes. It takes a source shard only when the shard's size and sha256 match turboderp's LFS record. It moves the routed experts of layers 4-71 into one file, experts.bin, with one record for each expert. It keeps every other tensor of each shard in a matching -nonexpert shard, and it puts the MTP layer in mtp-nonbank.safetensors. After it writes a shard, it reads every record back and compares it byte for byte with the source. The receipts in each branch record each check. Each branch's card lists every file.

The receipts do not record a hash of the script that ran. 2.10bpw ran from a copy that is byte-identical to commit 899fe84. 2.60bpw ran earlier, from 13:52 to 14:06 UTC on 2026-11-06, from uncommitted copies. They already wrote the classic record layout of commit aede45a and they ran before 899fe84 existed. 899fe84 changes only the text of the receipt's description. 2.60bpw's receipt file was written again 22 s after its finalize step, and its description has 899fe84's text. We checked some parts of the output that the script's code decides. The non-expert tensors have the same names, shapes and types in both branches. The 23 -nonexpert shard headers of the two packs are byte-identical, and their manifests and receipts have the same structure. We did not run 899fe84 again on 2.60bpw to compare every byte.

Credits

inclusionAI made Ling-3.1-flash and released it under the MIT license: inclusionAI/Ling-3.1-flash.

turboderp made the EXL3 quantizations that these packs hold, turboderp/Ling-3.1-flash-exl3. He also created exllamav3 and the EXL3 format, which is based on QTIP by Cornell RelaxML.

The server is mlx-serve by David Dalcu, with its bailing_hybrid Ling model, its plugin kit and its expert stream. The quant plugin turboexl3_ling is built on turboexl3. The repacks are by David Tai.

License

inclusionAI released Ling-3.1-flash under the MIT license, and turboderp's repository uses the same license. Neither repository has a LICENSE file, so the LICENSE file here is the standard MIT text with inclusionAI as the copyright holder.

Terms

Term	Meaning
EXL3	The quantization format of turboderp's exllamav3
bpw, K	Bits per weight. K is the bits per weight of one EXL3 tensor.
Routed expert	One of the 512 small feed-forward networks of an MoE layer; the router picks 8 for each token
Trunk	Every weight that is not a routed expert of layers 4-71 or the MTP layer
MTP layer	The multi-token prediction layer (layer 72), which drafts tokens for the main model to check
Prefill, decode	Prefill processes the prompt. Decode makes the reply, one round of drafts and checks at a time.
Bank, slot row	The GPU memory that holds routed experts read from the SSD. A slot row holds one expert of each routed layer.
Bill	mlx-serve's upper-bound estimate of a phase's memory
Fit, bank-fit.json	Measured memory peaks of one request, which replace the bills for one build and ceiling
Ceiling	The GPU memory that mlx-serve plans to, set by MLX_SERVE_GPU_CEILING_MB
Standard test	Our speed test: one 18,380-token coding prompt, a greedy reply of up to 1,024 tokens, 4 runs per arm
A/B, arm	A test that compares two builds; each build is one arm
NAX	The matrix units of the M5 GPU
bailing_hybrid	mlx-serve's model type for Ling
Claim rule	The test that a plugin uses to decide which packs it serves
TTFT	Time to first token, from the request to the first token of the reply
GB, GiB	GB is 10^9 bytes. GiB is 2^30 bytes.