OpensourceWTF/GLM-5.3-Flash-streaming-repack-exl3

GLM-5.3-Flash, streaming EXL3 packs of turboderp's self-calibrated quants, for mlx-serve

"We" is OpensourceWTF (David Tai). This card was written on 2026-11-14.

Each branch of this repo holds one of turboderp's self-calibrated EXL3 quantizations of GLM-5.3-Flash. We repacked each one so that mlx-serve can stream its routed experts from the SSD on Apple silicon. turboderp made the quantizations with exllamav3 and publishes them in turboderp/GLM-5.3-Flash-exl3. The quantizations are his. We did not quantize again and did not change any weight byte, so please credit turboderp when you use them.

mlx-serve's built-in glm5_next model serves the packs. An outside quant plugin, turboexl3_glm53f, runs the EXL3 weights and streams the experts. Each bitrate is a branch, and main holds only this card and the license.

Branches

Branch	bpw	Branch size	KLD (turboderp's)	Source and credit	Status
sc_4.10bpw	4.10	167.85 GB	0.0288	turboderp's sc_4.10bpw	live
sc_3.10bpw	3.10	128.07 GB	0.0574	turboderp's sc_3.10bpw	live
sc_2.60bpw	2.60	109.04 GB	0.0929	turboderp's sc_2.60bpw	live
sc_2.10bpw	2.10	88.31 GB	0.1570	turboderp's sc_2.10bpw	live

• Sizes are in GB, where 1 GB is 10^9 bytes. A branch size is the sum of the branch's files.
• bpw is turboderp's bits per weight for the branch. His count leaves out the embeddings and the output head.
• KLD is turboderp's mean KL divergence against the original model, read from the chart on his card. Lower is closer to the original. He does not publish perplexity for these branches.
• "live" means that the branch has landed and passed our file-by-file check: the name, size and sha256 of every file against the pack folder that we uploaded, on 2026-11-14.

Which branch to use

Use sc_3.10bpw on a 128 GB M5 Mac. It is the only branch that we served and measured: about 925 tok/s prefill and 39.7 tok/s decode on a 16K-token prompt (see Speed). It is also the only branch with a bank fit, a measured memory plan (see Memory). The other three branches use mlx-serve's default memory plan, which keeps fewer experts in memory, and we have no results for them. sc_4.10bpw has the lowest KLD, but we did not load it. sc_2.10bpw and sc_2.60bpw are smaller and less accurate.

We tested only a Mac with 128 GB of memory. Besides the expert slots, the trunk and the MTP experts stay in memory: 11.0 GB in sc_2.10bpw, 12.4 GB in sc_2.60bpw and sc_3.10bpw, and 14.2 GB in sc_4.10bpw.

The branches carry turboderp's chat template of each branch. sc_4.10bpw's template (10,950 bytes) is newer than the other three (8,617 bytes). Our test prompt and the usage profiles used the sc_3.10bpw template.

Related repositories

Repo	What it holds
turboderp/GLM-5.3-Flash-exl3	turboderp's EXL3 quantizations, one branch per bitrate. It is the source of every branch here.
zai-org/GLM-5.3-Flash	Z.ai's original model.
OpenSourceWTF/mlx-serve-streaming-pack-converters, glm53f/	The conversion scripts that repack his branches into this layout, the pack format (LAYOUT.md) and the exact revisions that built these packs.
OpensourceWTF/GLM-5.3-Flash-streaming-repack-exl3-3.05bpw	Our earlier GLM-5.3-Flash pack, now a stub that points here. Its trunk was FP8 and BF16 tensors that we made from another release, and its MTP experts were NVFP4, so turboderp's KLD does not apply to it. It is not a branch here.

How a pack works

GLM-5.3-Flash is Z.ai's mixture-of-experts (MoE) model. Each of its 42 MoE layers has 288 routed experts (small feed-forward networks), and a router picks 8 of them for each token. The routed experts are most of each pack. The server keeps only part of them in memory and reads the others from the SSD when a token needs them.

A pack is one of turboderp's branches with its routed experts moved out of the safetensors shards:

• experts.bin holds the routed experts of MoE layers 3 to 44, one record per expert. A record is the expert's gate, up and down projections, in turboderp's bytes.
• mtp-experts.bin holds the 288 experts of the MTP layer (layer 45), in the same record format. MTP (multi-token prediction) is the extra layer that proposes draft tokens. These experts stay in memory.
• Manifest files give the offset, size and sha256 of each record.
• Every other tensor stays in safetensors shards with turboderp's names, dtypes and bytes. We call these tensors the trunk. The trunk stays in memory.

Every file in a branch

File	What it is	Source
experts.bin	The routed experts of MoE layers 3 to 44: 42 layers x 288 records.	turboderp's bytes, in records
expert-manifest-v2.json, expert-manifest.json	The record table of experts.bin (offset, size and sha256 of each record, bits per weight of each layer), in the v2 and the older v1 form. Its parity.all_pass records the converter's checks.	ours
mtp-experts.bin, mtp-expert-manifest-v2.json, mtp-expert-manifest.json	The 288 experts of the MTP layer and their record tables.	turboderp's bytes, in records
model-000NN-of-000MM-nonexpert.safetensors	The trunk: one shard for each of turboderp's shards NN, with every tensor of it that is not a routed or MTP expert.	turboderp's tensors, in his order
model.safetensors.index.json	The index of the trunk shards.	ours
config.json	turboderp's file, plus the expert_streaming block that names the manifests and the source commit. The converter also changed its indentation.	turboderp's, with one block added
quantization_config.json	exllamav3's record of each tensor's bits.	turboderp's, unchanged
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, processor_config.json, LICENSE	The tokenizer, chat template and settings.	turboderp's, unchanged
expert-usage-v1.json, expert-usage-decode-v1.json	Expert-usage profiles. At load, the slots hold the experts that a profile ranks highest. See Limits.	ours
bank-fit.json (sc_3.10bpw only)	The bank fit. See Memory.	ours
TD-REPACK-RECEIPT.json	The source repo, branch and commit, the bits per weight of each layer, and the sizes and sha256 of the two record files.	ours
receipts/	The converter's check record for each source shard, its plan and state, and the source index and headers.	ours
manifest.sha256	The sha256 of every file in the branch except README.md and itself.	ours
README.md	The branch's card.	ours
.gitattributes	Hugging Face's storage settings for the repo.	Hugging Face

On 2026-11-14 we rewrote TD-REPACK-RECEIPT.json and manifest.sha256 to describe the usage profiles that the branches hold. The converter's versions named an earlier profile. No weight file changed.

Memory: the bank and the bank fit

The bank is the set of memory slots that hold routed experts, with the same number of slots in each layer. When a token routes to an expert that is not in a slot, the server reads its record from the SSD into a slot. More slots mean fewer SSD reads and faster decode.

When the bank opens, glm5_next decides how many slots each layer gets in the prompt phase and in the decode phase. It plans against the memory ceiling: the limit on the whole Mac's non-file memory, not on the server alone. Our tests set the ceiling to 118.1 GB (MLX_SERVE_GPU_CEILING_MB=112640, 110 GiB). glm5_next plans in one of two ways.

• With bank-fit.json (sc_3.10bpw only). The file holds memory peaks that we measured in one standard request (window g53fin_s1, 2026-11-13). glm5_next plans each phase so that the Mac's non-file memory peaks about 1 GiB under the ceiling. It applies the file only when the file's ceiling equals the server's ceiling, that is, only with MLX_SERVE_GPU_CEILING_MB=112640.
• Without the file (the other three branches, or another ceiling). glm5_next uses the bills. A bill is mlx-serve's upper bound on the memory that a phase of one request uses. The bank gets each phase's room under the ceiling, less the memory in use when the bank opens. A bill is an upper bound, so the bank gets fewer slots than the memory allows.

On sc_3.10bpw, the bills gave 147 prompt and 192 decode slots a layer, and that request decoded at 31.6 tok/s (one request, not an A/B). The bank fit gave 188 and 209 slots in our final speed test, which decoded at 39.73 tok/s.

A --ctx-size above the file's evidence request (17,411 tokens) makes the bank smaller: glm5_next takes the bills' growth from the evidence length to --ctx-size out of the bank. Our tests used --ctx-size 32768.

The bank fit's fields and rules

Field of sc_3.10bpw's bank-fit.json	Value	Meaning
format	mlx-serve-bank-fit-v2	The file format.
ceiling_bytes	118,111,600,640	The ceiling that the file applies at (MLX_SERVE_GPU_CEILING_MB=112640).
cell_tokens	17,408	mlx-serve's fixed reference request: a 16,384-token prompt and a 1,024-token reply.
evidence_tokens	17,411	The measured request: our 16,387-token prompt and 1,024 output tokens.
base_prompt_bytes	38,068,233,600	The Mac's peak non-file memory in the prompt phase, less the bank's bytes.
base_decode_bytes	31,743,548,992	The same in the decode phase.
rest_bytes	12,880,707,584	The non-file memory outside the server when the bank opened.

The file's sha256 starts cd36f4225219. The evidence request ran on mlx-serve a8c62ec3 with plugin 918a852.

glm5_next logs glm5_next: the bank fits by bank-fit.json (...) when it applies the file. Otherwise it logs glm5_next: bank-fit.json not applied (<reason>): the bank fits by the bills. The reasons are:

• BankFitOtherCeiling: the server's ceiling is not the file's.
• BankFitOtherCell: the server's reference request is not the file's.
• BankFitV1Rest: the file has the earlier format mlx-serve-bank-fit-v1, which the server refuses.
• BankFitFormat, BankFitInvalid: the file is not a v2 bank fit, or a field is missing.

If the file cannot be read, the log says glm5_next: bank-fit.json does not read (<reason>): the bank fits by the bills.

The code is bankFit in src/plugins/glm5_next/trunk.zig and parseBankFit in src/sdk/quant/outside_model.zig (mlx-serve 7b13376d). We have not published the tool that writes the file.

How to download and run

Requirements

• A Mac with an M5-class GPU, which has neural accelerators (NAX). On any other GPU the plugin refuses to load the pack.
• macOS 26.2 or later, Xcode 26.2 or later with its Metal toolchain, and Homebrew, to build mlx-serve.
• mlx-serve built from source at main 7b13376d, the commit that the plugin's kit pins. Later commits may need a newer plugin commit. 7b13376d includes ddalcu/mlx-serve#1192 (merged 2026-11-13), which lets glm5_next serve a streaming pack through an outside quant plugin. No mlx-serve release has it yet.
• The plugin turboexl3_glm53f 0.4.0, built from source. It is in CommunityServe/plugins#51, which stacks on CommunityServe/plugins#52. Neither is merged yet.
• Free disk for the branch, 88 to 168 GB. Keep the pack on the internal SSD. We did not test external drives.
• The Hugging Face CLI hf, which comes with huggingface_hub (pip install -U huggingface_hub).

A streaming pack serves text only, one request at a time: mlx-serve does not use the vision tower on this path, and it decodes one request at a time. We tested only sc_3.10bpw, on one Apple M5 Max with 128 GB of unified memory. The packs do not load in exllamav3, transformers, vLLM, SGLang or mlx-lm. For exllamav3, use turboderp's repository.

To reproduce our speed numbers, build mlx-serve from ddalcu/mlx-serve#1196, an open draft that adds GLM speed changes to glm5_next. Our numbers come from mlx-serve a8c62ec3, which has the same files as #1196's commit a9801c0a, with plugin 918a852. In #51 that plugin is commit d50a08b (the same plugin files), and #51's head d1349428 adds only a build guard. #1196's head, 87633f62, also holds a merge of main 7b13376d. It passed its tests, but we did not measure its speed. We did not measure main without #1196. The bank fit was measured with #1196, whose split prompt pass uses less memory in the prompt phase than main. On main without #1196 the bank fit may plan too many prompt slots. We did not test that, so use #1196 with the bank fit.

Download a branch

Pass the branch name to --revision. This downloads sc_3.10bpw only:

hf download OpensourceWTF/GLM-5.3-Flash-streaming-repack-exl3 --revision sc_3.10bpw \
  --local-dir ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw

Download a branch only when its status is "live".

Check the download

In the pack folder, check every file against manifest.sha256. Each line must end with OK:

cd ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw
shasum -a 256 -c manifest.sha256

The converter's own checks are in the branch. These commands print True, True and the number of source shards whose receipt passed (22 in sc_3.10bpw):

python3 -c "import json; print(json.load(open('expert-manifest-v2.json'))['parity']['all_pass'])"
python3 -c "import json; print(json.load(open('mtp-expert-manifest-v2.json'))['parity']['all_pass'])"
python3 -c "import glob, json; print(sum(json.load(open(f))['pass'] for f in glob.glob('receipts/model-*.json')))"

To compare the pack with turboderp's files on Hugging Face without the receipts, run python3 spotcheck_pack.py <pack folder> 8 from the converter. It compares random records and trunk tensors with his bytes.

Build mlx-serve

These are the "Server only" steps of mlx-serve's docs/building.md, plus the plugin kit:

git clone --recurse-submodules https://github.com/ddalcu/mlx-serve && cd mlx-serve
# To reproduce our numbers, use #1196's branch:
#   git fetch origin pull/1196/head:glm5_next-core-levers && git checkout glm5_next-core-levers
#   git submodule update --init
brew bundle install --file=Brewfile
./scripts/fetch-zig.sh && export PATH="$PWD/.zig-toolchain:$PATH"
./scripts/fetch-llama.sh && ./scripts/build-mlx.sh
zig build -Doptimize=ReleaseFast
zig build kit -Dversion=26.11.3
export PATH="$PWD/zig-out/bin:$PATH"   # the server, zig-out/bin/mlx-serve
MLXS="$PWD"                            # this checkout, for the plugin's kit
cd ..

The kit is in zig-out/kit/mlx-serve-plugin-kit-26.11.3/. The plugins repo expects kit version 26.11.3 (its KIT_VERSION file). The server's own version string can differ, and nothing checks it against the plugin's requires_mlx_serve. If xcrun -sdk macosx metal --version fails, run xcodebuild -downloadComponent MetalToolchain first.

Build and install the plugin

Build the plugin against the kit of the same mlx-serve checkout. The server refuses a plugin that was built against another MLX version.

git clone https://github.com/CommunityServe/plugins && cd plugins
git fetch origin pull/51/head:turboexl3_glm53f && git checkout turboexl3_glm53f
MLX_SERVE_KIT="$MLXS/zig-out/kit/mlx-serve-plugin-kit-26.11.3" scripts/install.sh turboexl3_glm53f
mlx-serve plugin check plugins/turboexl3_glm53f --model ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw

scripts/install.sh copies the kit into the checkout's .kit/ folder, runs the plugin's tests, builds it and installs it into ~/.mlx-serve/plugins/. It reuses .kit/ on later runs. After you build mlx-serve again, or switch between main and #1196, delete .kit/ (rm -rf .kit) before you run install.sh again.

Then turn plugins on. mlx-serve loads no plugin unless ~/.mlx-serve/plugins.json holds "enabled": true. This command writes that file, and replaces any file that is there:

printf '{"disabled": [], "enabled": true}\n' > ~/.mlx-serve/plugins.json

Start the server

These are the settings of our tests on a 128 GB Mac:

sudo sysctl iogpu.wired_limit_mb=114688
MLX_SERVE_GPU_CEILING_MB=112640 MLX_SERVE_CACHE_LIMIT=2147483648 \
mlx-serve --model ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw --serve --host 127.0.0.1 --port 8080 \
  --ctx-size 32768 --prefill-chunk 16384 --wired-margin-gib 2 --mtp-min-depth 2 --mtp-max-depth 2

Setting	What it does
iogpu.wired_limit_mb=114688	The macOS limit on GPU-wired memory (112 GiB). Our test Mac had this value. The setting lasts until a restart.
MLX_SERVE_GPU_CEILING_MB=112640	The memory ceiling, 110 GiB. Set it only on a 128 GB Mac: the server plans with this value in place of the Mac's limit. sc_3.10bpw's bank fit applies only at this value.
MLX_SERVE_CACHE_LIMIT=2147483648	Caps MLX's buffer cache at 2 GiB.
--ctx-size 32768	The longest context. A larger value takes memory from the bank.
--prefill-chunk 16384	The most prompt tokens in one forward pass.
--wired-margin-gib 2	How far under iogpu.wired_limit_mb a memory plan may reach.
--mtp-min-depth 2 --mtp-max-depth 2	Fixes the MTP draft at 2 tokens a round.

On a Mac with less memory, leave MLX_SERVE_GPU_CEILING_MB and the sysctl unset. The server then plans with the Mac's own limit and the bills. We did not test that.

The load log shows these lines when the pack loads:

• [diag] GPU ceiling forced to 112640 MB (MLX_SERVE_GPU_CEILING_MB): expected with that setting.
• [plugins] loaded turboexl3_glm53f (quant, sha256 ...): the plugin loaded.
• glm5_next: the bank fits by bank-fit.json (...): the bank fit applies (sc_3.10bpw only).
• glm5_next: streamed experts, ...: N prompt / M decode slot rows per layer: the slots that each layer got.

Options

The plugin reads 20 TURBOEXL3_GLM53F_* switches at load, and #1196 adds MLX_SERVE_* switches to glm5_next. With mlx-serve built from #1196's branch, the defaults give the configuration of the first row in Speed. These switches are off by default:

Switch	What it does	Where it acts
MLX_SERVE_KDA_PROMPT_CHUNK=1 (#1196)	Runs the KDA prompt recurrence in chunks of 16 tokens.	GPUs with fewer than 64 cores
MLX_SERVE_KDA_PROMPT_PERCORE=1 (#1196)	Runs the KDA prompt recurrence on the per-core kernel at up to two threadgroups a core.	GPUs with 32 to 63 cores
MLX_SERVE_GLM_DSA_PROMPT_ONEP=1 (#1196)	Rounds the DSA prompt attention's softmax weights to fp16 once.	NAX GPUs; DSA prompt passes of 17 or more tokens
TURBOEXL3_GLM53F_PF_INT8=1 to 4 (plugin)	Runs prefill matrix products in int8.	M5-class GPUs

The three #1196 switches change the rounding of the result, so the greedy text can change. Speed gives their gains. With CHUNK on, a load fails with KdaChunkStream, KdaChunkLayout, KdaChunkBound or KdaChunkThreads if its checks fail. PF_INT8 changes the values that the model computes, and we did not measure it with this plugin. The plugin's README (in #51) and the body of #1196 list every switch.

Troubleshooting

Log line or error	Cause	What to do
[plugins] refused <path>: plugin was built against another MLX: rebuild it with this release's kit, or ... plugin manifest version is not this host's: rebuild it with this release's kit	The plugin was built against the kit of another mlx-serve build.	In the plugins checkout, run rm -rf .kit, then the install.sh command above with MLX_SERVE_KIT set to the kit of the server's checkout.
[plugins] refused <path>: has no install receipt: install it with mlx-serve plugin install``, or changed since it was installed (hash differs from its receipt)	The plugin file was copied in by hand, or rebuilt after install.	Run scripts/install.sh turboexl3_glm53f again.
[fetch-kit] cannot download ... from install.sh	No mlx-serve release has kit 26.11.3, and MLX_SERVE_KIT is not set.	Set MLX_SERVE_KIT as above.
glm5_next: a streaming pack (expert_streaming): its experts.bin bank is not streamed by this plugin, or StreamedBankUnclaimed, and no [plugins] loaded turboexl3_glm53f line	No plugin serves the pack: it is not installed, plugins are off, or the server ran with --no-plugins.	Check that ~/.mlx-serve/plugins.json holds "enabled": true, and do not pass --no-plugins.
turboexl3_glm53f: an M5 or newer GPU (NAX) is required	The GPU is not M5-class.	Use an M5-class Mac.
turboexl3_glm53f: the expert_streaming block is not the streaming repack's (LAYOUT.md)	The folder is not one of these packs, for example our old 3.05 pack.	Download a branch of this repo.
turboexl3_glm53f: the pack lacks its banks (experts.bin, mtp-experts.bin and their manifests), or glm5_next: a streaming pack without its experts.bin bank	The download is not complete.	Download the branch again and run the checks above.
turboexl3_glm53f: the codebook is not mul1	The pack's EXL3 codebook is not mul1.	Use a branch of this repo. All of them are mul1.
StreamedRowsDoNotFit	The bank does not get enough memory for its slots.	Close other apps, lower --ctx-size, or check MLX_SERVE_GPU_CEILING_MB.
glm5_next: bank-fit.json not applied (<reason>): the bank fits by the bills	See the reasons in Memory.	Use MLX_SERVE_GPU_CEILING_MB=112640 for sc_3.10bpw's file. Without the file the server still runs.
[turboexl3_glm53f] <switch> is set, but this kit's expert stream (kit.streamed) has no seam for it: the bank is refused (set it to 0)	TURBOEXL3_GLM53F_PROMPT_KEEP or ADOPT_PIECES is set to 1, and mlx-serve does not have their stream changes.	Unset the switch.

Quality

turboderp measured the KL divergence of each quantization against the original model, with exllamav3 on his files. His trace was self-generated and in-domain, 28,154 input and 45,470 output tokens. His chart marks the measurement's noise floor at 0.01378. The packs hold his quantized bytes in a different file layout, so his numbers describe the weights in these packs. They do not include mlx-serve's kernels.

The chart is turboderp's, linked from his card. EXL3-SC is his self-calibrated series, the source of these branches. EXL3 is his plain series.

We did not measure KL divergence against the original model. Our quality checks through mlx-serve compare one configuration with another on sc_3.10bpw:

• Each of #1196's three rounding-class switches moved the mean next-token loss on 8 code documents by less than 0.001 nats a token. Each switch's change was within 2 standard errors of the change from a reference switch (ONEP for the KDA switches, PERCORE for ONEP). The body of #1196 has the tables ("The floor tests").
• Against our earlier whole-model plugin (closed, see Earlier results), the core path scored the prompt tokens of 8 code documents 0.00063 nats a token worse (standard error 0.00031). The likely cause is rounding: the earlier plugin ran rounding-class routes. This test (g53cpnll3, 2026-11-12) ran on an earlier core build (mlx-serve 1e7a6d64, plugin 94e3221), before #1196's changes. #51's body has it.

We have no task-level test (such as HumanEval) on the core path.

Speed (sc_3.10bpw only)

All speed results come from one machine: an Apple M5 Max (40-core GPU) with 128 GB of unified memory, on macOS 27.2. The test is our standard 16K test: one 16,387-token coding prompt, then 1,024 output tokens, greedy, with MTP drafts at depth 2. An A/B runs 8 runs in the order A B B A, then B A A B. Each run starts a fresh server with the pack's files evicted from the page cache, and waits until the chip is below 40 C. Each value is the mean of the 4 runs of one arm.

Configuration	Prefill tok/s	Decode tok/s	TTFT s	Wall s	Spec-decode hit %	Peak GB
#1192 + #1196 at their defaults, with the bank fit	925.4	39.73	17.72	43.56	74.1	101.4
The same + MLX_SERVE_KDA_PROMPT_CHUNK=1	1,073.8	38.18	15.27	42.15	78.0	100.6
The same + MLX_SERVE_KDA_PROMPT_PERCORE=1	1,054.5	38.55	15.55	42.17	76.6	100.7
The same + MLX_SERVE_GLM_DSA_PROMPT_ONEP=1	933.4	38.48	17.57	44.26	77.1	100.5
Our earlier whole-model plugin, closed	1,236.9	40.73	13.26	38.46	69.2	104.3

• Sources: the first row is arm B of window g53fin_ab1 (2026-11-13). The last row is arm A of the same window. The three switch rows are the B arms of windows g53kp5ab_ch, g53kp5ab_pc (2026-11-13) and g53kp5ab_op (2026-11-14). A window is one scheduled test session on our machine.
• Prefill and decode come from the server's timings. TTFT and wall time come from the client. The spec-decode hit rate is the share of MTP draft tokens that the model accepted.
• Peak is the server process's peak memory footprint. The ceiling bounds the whole Mac's non-file memory, which also holds other processes: in the bank fit's evidence request, 12.9 GB was outside the server.
• The first row writes the same text in every run. Each rounding-class switch changes the text, and a different text decodes at a different speed. So compare those rows on prefill and TTFT only.
• Each switch A/B ran both arms on build da4ccf42, not on the first row's build a8c62ec3. Their A arms had the first row's switch settings and prefilled at 921.8 to 923.5 tok/s. Read each gain against its own A arm: CHUNK +16.3%, PERCORE +14.4%, ONEP +1.2%.
• The three switches were measured one at a time. Do not add their gains: PERCORE and CHUNK speed up the same step.
• We did not run a context sweep on the core path.

Per-run rows of g53fin_ab1

Window g53fin_ab1, 2026-11-13 15:19 UTC. A is our earlier whole-model plugin (969a2bc on our build kit7). B is glm5_next on mlx-serve a8c62ec3 (#1192 at f4ccce40 plus #1196's exact changes, the same files as #1196's commit a9801c0a), plugin 918a852 and the bank fit. B held 188 prompt and 209 decode slots a layer. The texts differ by design: A runs rounding-class routes. A wrote text c3c34622230749d9 and B wrote b1739c1005558f6f.

Run	Arm	Seed	Prefill tok/s	Decode tok/s	TTFT s	Wall s	Spec-decode hit %	Peak GB
1	A	1234	1,237.4	40.70	13.26	38.46	69.2	104.3
2	B	1234	927.9	39.80	17.67	43.47	74.4	101.4
3	B	4321	926.6	39.50	17.70	43.68	74.4	101.4
4	A	4321	1,236.9	40.80	13.26	38.45	69.2	104.3
5	B	1234	922.6	39.80	17.77	43.54	73.2	101.4
6	A	1234	1,235.6	40.70	13.27	38.49	69.2	104.3
7	A	4321	1,237.8	40.70	13.25	38.45	69.2	104.3
8	B	4321	924.5	39.80	17.74	43.54	74.4	101.4

Every arm used the server command in Start the server. Under greedy decoding the seed does not change the text. An exact change is one whose output bytes equal the default path's; B's text is the text of glm5_next with all of #1196's switches off.

The three switch A/Bs ran on 2026-11-13 and 2026-11-14 on build da4ccf42: #1196's exact changes, the three switches, and a fourth switch (FOLD) that was off and is not in #1196. They used plugin 918a852 and the same bank fit. Their per-run rows are in the body of #1196, in "Per-run rows of each A/B".

Earlier results

Before the core path, we served sc_3.10bpw with our own whole-model plugin, which carried its own copy of the model's forward. Its pull request, CommunityServe/plugins#48, is closed. On earlier builds of that plugin we ran a context sweep from 1K to 256K tokens, int8 prefill, HumanEval and a 32-document loss test. These results do not describe the core path. #48's body links the sweep report and every run.

Limits

• We measured only sc_3.10bpw, on one machine. We have no speed, memory or quality results for the other three branches, and we did not load sc_4.10bpw.
• #1196, #51 and #52 are open and not merged, and no mlx-serve release has #1192 yet.
• A streaming pack serves text only, and one request at a time.
• We recorded each expert-usage profile on one request of our 16K test prompt. So that prompt can start with more of its experts in memory than another prompt does. We did not measure how large this effect is.
• The speed tests used one prompt size and one request at a time.

How the packs were made

Our converter, convert_glm_sc.py in the converter repository, reads each of turboderp's branches at a pinned commit from Hugging Face, with HTTP range requests. It writes the routed and MTP experts as records, and every other tensor into new shards in the source order. It changes no weight byte.

For each source shard, the converter compares the sha256 of the streamed bytes with the shard's LFS hash at the pinned commit. After it writes the pack, it reads every record and every trunk tensor back from disk. It compares each with the sha256 of its source tensor. It writes a receipt for each shard, and it finishes a pack only when every receipt passes. In all four packs, expert-manifest-v2.json and mtp-expert-manifest-v2.json record parity.all_pass: true.

Credits

• turboderp made exllamav3, the EXL3 format and the self-calibrated quantizations that these packs contain: turboderp/GLM-5.3-Flash-exl3. EXL3 is based on QTIP by Cornell RelaxML.
• Z.ai made GLM-5.3-Flash: zai-org/GLM-5.3-Flash.
• ddalcu maintains mlx-serve, which has the glm5_next model.
• turboexl3 has the Metal kernels that the plugin runs for the EXL3 weights.

License

MIT, as turboderp's repository. The LICENSE file is turboderp's, unchanged, and every branch carries it. The code (mlx-serve, the plugin, turboexl3 and exllamav3) has its own licenses.

Terms

Term	Meaning
bpw	Bits per weight. turboderp's count leaves out the embeddings and the output head.
SC	Self-calibrated: turboderp's name for these four branches. In them the bits per weight differ by layer and tensor type.
Routed expert	One of the 288 small networks in a MoE layer. The router picks 8 for each token.
Record	The bytes of one expert of one layer in experts.bin or mtp-experts.bin.
Trunk	Every tensor that is not a routed or MTP expert: attention, shared experts, dense layers, embeddings, LM head, vision tower.
Slot	A place in memory that holds one expert record. The log calls it a slot row.
Bank	The set of slots for the routed experts.
Bill	mlx-serve's upper bound on the memory that a phase of one request uses.
Bank fit	The slot plan from measured peaks (bank-fit.json) in place of the bills.
Ceiling	The limit on the whole Mac's non-file memory that the server plans against (MLX_SERVE_GPU_CEILING_MB).
Core path	mlx-serve's built-in glm5_next with the quant plugin. Our earlier whole-model plugin ran its own copy of the model instead.
Exact change	A change whose output bytes equal the default path's.
Rounding-class switch	A switch that computes the same math with different rounding. The result is close to the default's but not the same bytes.
KDA, DSA	The model's two kinds of attention layer: 34 linear-attention layers with a recurrent state, and 11 sparse-attention layers.
MTP	Multi-token prediction: the model's extra layer that proposes draft tokens, which the main model then checks.
Prefill	Processing the prompt, before the first output token.
TTFT	Time to first token, from the request to the first output token, on a loaded model.
KLD	Mean KL divergence between the original model's and the quantized model's next-token distributions. Lower is closer.
Window	One scheduled test session on our machine. Its ID (for example g53fin_ab1) names its logs.