WEX4 Expert Containers
Expert weights of three Mixture-of-Experts AI models (Qwen3.8-Flash-Next, DeepSeek V4.1 REAP-192 and Sage-S2.1-Flash), stored in WEX4 files.
Storing the weights in these files does not change the value of any weight. It is not a quantization. WEX4 is the file layout. WNR3 is optional compression inside it, like ZIP. Unpacking gives back the same weights: the very same bytes for the two Qwen files, and the very same numbers for the DeepSeek file (a zero written as "-0" comes back written as "+0"; same number, explained below).
The weights are 4-bit, and making them 4-bit did lose some precision. That happened before the weights reached these files: AMD did it for Qwen, DeepSeek did it for its own model, and we did it ourselves for Sage. The table below says who did what.
In plain words
What is in these files. A Mixture-of-Experts model is split into hundreds of small blocks of weights called experts, and only a few of them work on each word. When a model is too big for a computer's memory, a program keeps some experts in memory and reads the others from the SSD the moment they are needed. These files hold those experts, laid out so that each one comes off the disk in a single read (except the Sage int4 file, which is a packed copy of a whole set of model files).
Think of a ZIP file. When you zip a document and unzip it later, you get the same document back, every letter in place; the ZIP was only a smaller way to keep it. WNR3 does that for 4-bit expert weights. Files with .nr3 in their name use it and are 12-15% smaller. The other files are not compressed by WNR3: they hold the weights' original bytes, one expert per slot.
Where precision was lost: before these files. To quantize is to round numbers to fewer bits so that they take less space; the rounding cannot be undone. The weights here are 4-bit numbers (in formats called MXFP4 and int4). Turning a model's original 16-bit or 8-bit weights into 4-bit numbers is a quantization, and it does lose some precision. That step happened before the weights reached these files. Two of the files hold a 4-bit version that we made ourselves from the official 8-bit Sage release. Making them lost some precision, and we have not measured how much.
| File | What changed the weights before they reached our file, losing precision, and who did it | What our file does to them |
|---|---|---|
qwen38-mxfp4/q38_experts.wex |
AMD rounded the experts of Qwen3.8-Flash-Next to 4 bits (MXFP4) and published them as amd/Qwen3.8-Flash-Next-Quark-MXFP4 | Not compressed. Keeps AMD's bytes as they are (compared with AMD's files when it was built: every expert of the first and the last layer, and the first and last expert of every layer) |
qwen38-mxfp4/q38_experts.nr3.wex |
the same AMD 4-bit weights | Compressed to 88% of the uncompressed file. Unpacking gives exactly the same bytes |
v41-reap/v4_flash_experts.nr3.wex |
DeepSeek publishes the experts of DeepSeek-V4.1-Flash-1017 already in 4 bits (MXFP4). Then a third party removed half of the experts and published the rest as patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-MXFP4 ("REAP" pruning: 192 of 384 experts kept in each layer, the kept ones copied unchanged). Removing experts loses quality | Compressed to 85% of the size. Every weight is the same number (one detail about zero, see below) |
Sage-S2.1-Flash-int4/Sage-S2.1-Flash-int4.wex4 |
Shanghai AI Laboratory published Sage-S2.1-Flash and an official 8-bit version, internlm/Sage-S2.1-Flash-FP8. We rounded its experts to 4-bit integers (int4) and turned its other 8-bit tensors into 16-bit BF16, which can round too. Lossy; we have not measured the quality | Keeps our int4 files unchanged. A small part is compressed ZIP-style (WBP1), and that is lossless too |
Sage-S2.1-Flash-MXFP4/Sage-S2.1-Flash-MXFP4.wex |
the same official 8-bit version; we rounded its experts to MXFP4. Lossy; we have not measured the quality | Not compressed. Uses the older WEX1 layout, which has no fingerprints inside; the manifest file next to it has them |
Sage-S2.1-Flash-MXFP4/Sage-S2.1-Flash-MXFP4.nr3.wex |
exactly the same Sage MXFP4 weights as the file above | Compressed to 83% of the older file, in the newer WEX4 layout with a fingerprint for every expert. Unpacking gives exactly the same bytes: all 12,096 experts were unpacked and compared byte for byte with the source files, 0 differences |
Is this a quantization? No.
Does WEX4 or WNR3 reduce the precision of the weights? No. Quantization rounds every weight to a number with fewer bits, so the numbers change. WEX4 and WNR3 round nothing: unpack a stored expert and you have the weights that were put in.
Then why is a WNR3 file smaller? For the same reason a ZIP is smaller. In real weights some 4-bit values occur much more often than others. WNR3 gives the common values short codes and the rare values long codes. It uses a table that says how common each value usually is; if the table guesses wrong, the file only gets bigger, never different. Nothing is generated or predicted by an AI: it is plain arithmetic, and a stored expert that would not get smaller is kept uncompressed.
So why does it say "MXFP4" and "4-bit" everywhere? Because the weights were already 4-bit when we got them (Qwen, DeepSeek), or when we made them (Sage). We keep them exactly in that form.
Is anything lossy, then? Yes, the steps in the table, all of them before our files: DeepSeek's own 4-bit release, AMD's 4-bit rounding of Qwen, the REAP pruning of DeepSeek, and our own rounding of Sage (both files, including the 8-bit to BF16 step of the int4 file). Nothing is lossy in the WEX4 layout or in the WNR3 compression.
What does "value-exact" mean for the DeepSeek file? In 4-bit floating point, zero can be written two ways, +0 and -0, and both are the same number. It is like writing 0 or -0 on paper: the amount is the same. To save space (85% of the size instead of about 88%), WNR3 writes every -0 as +0. Every weight unpacks to the same number; only the way some zeros are written differs from the source files. We ran the model from this file and from a byte-exact copy: its output scores were identical to the last bit on 7 short prompts (64 tokens each), and a long 1,886-token conversation turn produced the same text (report). The two Qwen files are byte-exact.
Why are there two files of the same Qwen experts? The compressed file is 7.7 GB smaller, but unpacking costs computer work. On the test machine (one run per case, RTX 5090, earlier copies of the same records) that work made generation 19-29% slower, for example 9.48 instead of 11.48 tokens per second (report). The program gets the same bytes from either file.
The words, one sentence each
- Lossy: some detail is rounded off or dropped and cannot be recovered.
- Quantization (requantize): rounding weights to fewer bits to make them smaller; it is lossy. To requantize is to do it again to weights that were already rounded once, for example from 8 bits to 4.
- Lossless: unpacking gives back exactly what was packed, with nothing rounded or dropped.
- Byte-exact: every byte is the same; a byte-by-byte comparison finds zero differences.
- Value-exact: every weight unpacks to the same number; only the way zero is written (+0 or -0) may differ.
- Token: a piece of a word; models read and write text in tokens.
- SHA-256 per record: each stored expert has a fingerprint (SHA-256) in the file's table of contents; if even one bit of the stored data changes, the fingerprint no longer matches, so a damaged download is caught. This proves the file is intact; proving that unpacking is exact is a separate check (below).
- Root: one fingerprint for the whole file; if any part of the file changed, the root changes too.
- Content digest: one fingerprint of all the unpacked weights; two files that unpack to the same weights give the same content digest, compressed or not.
Check it yourself
Each check answers a different question. The tools are open source in KairaCode/Wex-WRN2 (Apache-2.0).
1. Is my download intact? Nothing to build:
sha256sum q38_experts.nr3.wex # must equal the SHA-256 that Hugging Face lists for the file (Integrity table below)
2. Is every stored expert undamaged? wex4_verify checks every stored record against its fingerprint and prints the root. It does not unpack anything. Needs git and gcc with OpenMP:
git clone https://github.com/KairaCode/Wex-WRN2 && cd Wex-WRN2
gcc -O2 -fopenmp c/tools/wex4_verify.c -o wex4_verify
./wex4_verify v4_flash_experts.nr3.wex # about 150 s from an NVMe SSD, 260 s from a SATA SSD (8 threads, i7-7700K)
3. Does unpacking give exactly the same weights? wex4.py verify unpacks every record and prints the content digest. Needs Python 3 and the helper library that check_all.sh builds (it is named .dll on every system):
bash scripts/check_all.sh
python -I wex4/wex4.py verify q38_experts.wex --dll wex4/out/wex4_tools.dll
python -I wex4/wex4.py verify q38_experts.nr3.wex --dll wex4/out/wex4_tools.dll
For the two Qwen files: if both print the same content digest, 0d752bbf2d169b611ac79b005f07d0b9152c0591b23b65cbd12e06ea8300ec77 for the uncompressed file, then every expert of the compressed file unpacked to exactly the same bytes. The DeepSeek file has no uncompressed twin here; its content digest when we checked it was fb085e127ba980214f106313e9af578b18e5ebac4cde7ecb9730b1a0f78ae512.
4. The Sage files.
- Sage int4:
python -I wex4/wex4_convert.py unpack Sage-S2.1-Flash-int4.wex4 OUT_DIR --dll wex4/out/wex4_tools.dllrebuilds the 62 safetensors files (about 195 GB of free disk needed) and checks each one against the SHA-256 recorded inside the file. That proves you get back our int4 files exactly. - Sage MXFP4 (older layout, no fingerprints inside):
python -I c/tools/repack_q38_mxfp4_wex.py --verify Sage-S2.1-Flash-MXFP4.wex(needs numpy) re-hashes every layer againstSage-S2.1-Flash-MXFP4.wex.manifest.json.
What you cannot check yet. No published command compares these files with the original upstream downloads (AMD's Qwen files, the REAP-192 files, the official Sage FP8 files). The checks above prove that the files are intact and that unpacking is exact; the link to the upstream files rests on the build-time checks described in the Integrity section.
The rest of this card is technical detail.
Files
| Path | Model and layout | Records | Size | Upstream license |
|---|---|---|---|---|
qwen38-mxfp4/q38_experts.wex |
Qwen3.8-Flash-Next routed experts, AMD-Quark MXFP4; WEX4, RAW records | 24,576 (48 layers Γ 512 experts) | 64,251,047,936 B (64.25 GB) | Qwen Community License 1.0 |
qwen38-mxfp4/q38_experts.nr3.wex |
the same experts; WEX4, WNR3 records, byte-exact | 24,576 (48 Γ 512) | 56,558,764,032 B (56.56 GB) | Qwen Community License 1.0 |
v41-reap/v4_flash_experts.nr3.wex |
DeepSeek-V4.1-Flash-1017 REAP-192 routed experts, MXFP4; WEX4, WNR3 records with canonical zero (value-exact) | 7,680 (40 Γ 192) | 123,312,910,336 B (123.31 GB) | MIT |
Sage-S2.1-Flash-int4/Sage-S2.1-Flash-int4.wex4 |
our int4 requantization of the 62 model shards of Sage-S2.1-Flash-FP8 (see below); WEX4 holding 62 safetensors files (ST_SOURCES) | 75,273 (74,304 RAW, 969 WBP1); 75,922 tensors, of which 12,384 routed experts (43 Γ 288) | 187,770,384,384 B (187.77 GB) | MIT |
Sage-S2.1-Flash-MXFP4/Sage-S2.1-Flash-MXFP4.wex |
Sage-S2.1-Flash routed experts of layers 3-44 in our MXFP4 requantization; WEX1 (older layout: no index fingerprints) | 12,096 (42 Γ 288), 13,369,344 B each | 161,716,523,008 B (161.72 GB) | MIT |
Sage-S2.1-Flash-MXFP4/Sage-S2.1-Flash-MXFP4.wex.manifest.json |
SHA-256 of each layer's records and of the header page and table of the WEX1 file | β | 10,924 B | β |
Sage-S2.1-Flash-MXFP4/Sage-S2.1-Flash-MXFP4.nr3.wex |
the same Sage-S2.1-Flash routed experts (layers 3-44, our MXFP4 requantization); WEX4, WNR3 records with canonical zero (byte-exact here: the experts contain no -0) | 12,096 (42 Γ 288) | 134,545,358,848 B (134.55 GB) | MIT |
Sage-S2.1-Flash-MXFP4/Sage-S2.1-Flash-MXFP4.nr3.wex.json |
build record of the WNR3 file (codec settings, per-layer hashes) | β | 6,216,021 B | β |
provenance/q38_experts.wex.verified.pre-wex4 |
SHA-256 of the WEX1 copy the Qwen RAW file was upgraded from | β | 123 B | β |
provenance/v4_flash_experts.nr3.wex.verified.pre-wex4 |
build and read-back note of the DeepSeek file | β | 177 B | β |
The Sage int4 file holds the language model (including the MTP layer 45) and the vision tower model.visual (347 tensors): experts in int4 (one F32 scale per 64 weights), the FP8 non-expert tensors dequantized to BF16 (874 tensors), the F32 tensors kept as F32 (744). It does not hold the 9 projectors/β¦/projector.safetensors, config.json, the tokenizer or the chat template of internlm/Sage-S2.1-Flash-FP8. The other files hold routed experts only: a runtime takes the rest of the model from the original checkpoint. None of them loads with transformers.from_pretrained().
WEX4 and WNR3 are separate layers:
- WEX4 is the container: one record per expert (or per tensor group), 4096-byte aligned so that an expert is one aligned read; a self-describing tensor index; a SHA-256 of the stored bytes of every record and blob; and a root, the SHA-256 of the header page, that names the whole file. WEX4 does not compress, except with the optional WBP1 codec, which is lossless byte-plane rANS for BF16, FP8, F32 and integer tensors.
- WNR3 is a lossless codec for MXFP4 expert records (FP4 E2M1 weights and E8M0 scales): context-modelled rANS entropy coding with a 59,408-byte table of value statistics (the WNM2 model, a count table, not a neural network) stored once in the file, a 23-bit hash in every stream, and a CPU decoder and an NVIDIA GPU decoder. With canonical zero it stores FP4 -0 as +0 (value-exact); without it, it is byte-exact.
Integrity
| File | WEX4 root (COLI_WEX_ROOT) |
SHA-256 of the whole file (as listed by Hugging Face) | Checked |
|---|---|---|---|
q38_experts.wex |
db5389f699abb49a164a9e73d925b5b629b0ce6de430a553388ba3e1cae5c0df |
9f6caa9a1e4e5ffe300108d3865ad0c4a246862c86ddaea2d3ee56b37e286bbf |
24,576 records, every hash matches the index; content digest 0d752bbf2d169b611ac79b005f07d0b9152c0591b23b65cbd12e06ea8300ec77. Against AMD's tensors: all 512 records of layers 0 and 47, and the first and last record of every layer, byte-identical when the file was built |
q38_experts.nr3.wex |
7d78e6ccfb8dee558a08371da86d275d53b7356d255d25ae4cac831dabd3e7f9 |
abddc4243b56a109d74973cee324b19296db0cf5b996f2e8a5585b798e29255b |
at build time, layers 0, 10, 23 and 47 decoded on the GPU and compared with the RAW records. An archive copy was later found damaged (11,666 records zeroed, old root cd47fa8bβ¦, not valid) and was repaired; on 2026-11-20 the repaired local copy passed a full check and all 48 layers decoded equal to the RAW file. The upload itself was not re-verified (only its SHA-256 is known) |
v4_flash_experts.nr3.wex |
f0f78a84d040a3b336f3abef3cf8b0a36c649deb38586c813e94d35dea3beeb1 |
203d55dc96f5c2fe16716cef2c767fa63c387e8cbbbe98991ecd4f7039c7e4b2 |
7,680 records decoded, every hash matches the index; content digest fb085e127ba980214f106313e9af578b18e5ebac4cde7ecb9730b1a0f78ae512 |
Sage-S2.1-Flash-int4.wex4 |
85246e738032d8734d27575448952f69f49308f8cb5caaba7b281b13ce25bf0d |
bbfa112154425fb73339ebd1b05d3d53f987b8374a2e5481ba5c1f3e8e3e0747 |
a SHA-256 for each of the 62 source files is inside; no full check of this file is on record, so run wex4_verify or unpack |
Sage-S2.1-Flash-MXFP4.wex |
none (WEX1) | cae23f8575f92f29be895ef460caf1d66d764a870389b6a234a1aff876949249 |
a local copy was re-hashed against this manifest (header page and table ef4b66cfcc72f539367eaffc64bbb18ddb442b90ecd06a63a5169bea8596a158) |
Sage-S2.1-Flash-MXFP4.nr3.wex |
35de35aa5766e8f4a7e4fa69c695d83d69648b6ceb6a7c4624f4bf20b1a689dd |
96d9124e3d4fd9fc1b2f3812f39d374e41b21ab91a4381fe7f2f9fe1bfc1436d |
wex4_verify: every blob and record matches the index; all 12,096 records decoded and compared byte for byte with the 62 source safetensors files (161.7 GB, 0 failures); every layer's records hash equal to the WEX1 manifest above. The root was read back from the hub after upload |
The root is the SHA-256 of the first 4,064 bytes of the header page. The record SHA-256 covers the stored bytes, so wex4_verify proves the file is intact, not that decoding is exact. The content digest is the SHA-256 over, for each tensor in table order, its name, dtype, shape, size and decoded bytes; it does not cover metadata, blobs or feature bits (WEX4-SPEC section 5). The WNR3 builder decodes every record on the CPU and refuses to write one that does not decode back exactly. A damaged WNR3 record is caught twice: by the stream hashes (0 of 2,231,336 wrong single-bit flips accepted on the GPU, 0 of 3,000 on the CPU, WNR3-VERIFY) and by the record's SHA-256 in the index.
Size
| File | Stored | Original bytes | Ratio |
|---|---|---|---|
| Qwen3.8, WNR3 file against the RAW file | 56,558,764,032 B | 64,251,047,936 B | 0.880 |
| DeepSeek V4.1 REAP-192, WNR3 with canonical zero, against the raw WEX1 container of the same experts | 123,312,910,336 B | 144,389,472,256 B | 0.854 |
| DeepSeek V4.1 REAP-192, WNR3 byte-exact (40 held-out experts, not a full file) | about 0.882 | ||
| Sage int4, against its 62 source safetensors files | 187,770,384,384 B | 194,727,333,920 B | 0.964 |
| Sage MXFP4, WNR3 file against the older WEX1 file of the same experts | 134,545,358,848 B | 161,716,523,008 B | 0.832 |
| Qwen3.8 RAW and Sage MXFP4 WEX1 | 1.000 (not compressed) |
Quality of the 4-bit weights (steps taken before the container)
The container adds no error, so these are the only quality questions, and they belong to the upstream steps:
- Qwen3.8, AMD's MXFP4. Close to the original model. AMD reports, against the original, among others: GSM8K 96.8 β 96.5, GPQA Diamond 91.67 β 91.67, AIME26 99.17 β 98.33, IFBench 83.27 β 82.67. These are 4 of AMD's 8 benchmarks; the largest drop is MMMU Pro vision, 78.38 β 77.40; the full table is on AMD's card. Our own check against the official FP8 weights, very close: minimum cosine 0.9926 over the checked matrices of all 48 layers, and on 9 prompts a mean gap of 0.018 nats/token on text and 0.013 on chat (report).
- DeepSeek V4.1, REAP-192 pruning. A measurable loss. The pruned model's author reports, teacher-forced against the unpruned model on 185,565 held-out tokens: top-1 agreement 0.798, KL on the top-32 support 0.44, NLL +0.27; multilingual top-1 0.52. The author states that the checkpoint failed its release gate as pre-registered and passed under an amendment made after the result (card).
- Sage-S2.1-Flash, our int4 and MXFP4. Not measured. The int4 file uses symmetric int4 with one F32 scale per 64 weights (the layout of colibri's
convert_glm53.py); the MXFP4 file uses FP4 E2M1 weights with one E8M0 scale per 32. Treat both as experimental, and use the official FP8 release where quality matters.
Use with colibri
The WEX4 series for colibri lives on the fork KairaCode/colibri (main) and is proposed upstream in JustVugg/colibri#2920. Everything is opt-in: with COLI_WEX_PATH unset an engine behaves as before. The Sage files are not part of that series.
# DeepSeek V4.1: trunk from the original checkpoint, routed experts from the container
export COLI_WEX_PATH=/models/v4_flash_experts.nr3.wex
export COLI_WEX_ROOT=f0f78a84d040a3b336f3abef3cf8b0a36c649deb38586c813e94d35dea3beeb1 # refuse any other file
export COLI_WEX_CUDA_LIB=/path/to/kaira_wex_cuda.so # optional: decode WNR3 on an NVIDIA GPU (make wex-cuda-helper)
WNR3 records decode on the CPU (a fast AVX2/NEON decoder, COLI_WEX_FAST_DECODE, on by default) or, when COLI_WEX_CUDA_LIB names the helper, on the GPU. COLI_WEX_VERIFY (on when a root is pinned) checks each record's SHA-256 the first time it is read.
Measured with deepseek_v41 on one desktop (i7-7700K 4C/8T, 32 GB RAM, RTX 3070 8 GB, Windows 10), the DeepSeek file on NVMe, trunk from the shards on a SATA SSD, 8 expert slots, prompt "The three primary colors of light are", 8 greedy tokens: the same tokens in every arm.
| Arm | 8 tokens | tok/s |
|---|---|---|
| shards only (no container) | 62.5-91.6 s | 0.09-0.13 |
| container, fast CPU decoder | 71.1 s | 0.11 |
| container, CUDA helper on the RTX 3070 | 35.2 s | 0.23 |
One DeepSeek record decodes in about 101 ms on one CPU thread with the fast decoder (742 ms with the reference decoder) and in about 6 ms on the RTX 3070.
Help wanted: runs on other machines (ARM, Apple silicon, other GPUs, 16 GB RAM, two NVMe drives) with the one-command bench c/tools/wex_bench.py: the list and how to report are in KairaCode/colibri#1.
Download
hf download Kaira666/Kaira-WEX4-Experts --local-dir WEX4-Experts # everything, about 594 GB
hf download Kaira666/Kaira-WEX4-Experts v41-reap/v4_flash_experts.nr3.wex --local-dir WEX4-Experts # one file
hf download Kaira666/Kaira-WEX4-Experts qwen38-mxfp4/q38_experts.nr3.wex --local-dir WEX4-Experts
Source and specifications
The format, the codecs, the specifications, conformance vectors, four independent readers (C, Python, Go, Rust) and the converters are open source under Apache-2.0 in KairaCode/Wex-WRN2:
- WEX4-SPEC and WNR3-SPEC
- Colibri integration guide
wex4/wex4_convert.py:packa safetensors checkpoint into WEX4 (RAW or WNR3 experts),unpackit back byte for byte
Licenses
The files here carry different terms, so no single license is asserted for the repository:
- Qwen3.8-Flash-Next expert weights (both Qwen files, quantized by AMD): Qwen Community License 1.0, the license of the base model and of AMD's release.
- DeepSeek-V4.1-Flash-1017 REAP-192 expert weights: MIT, Copyright (c) 2023 DeepSeek; the REAP-192 release is MIT too.
- Sage-S2.1-Flash weights (both Sage files, requantized by us): MIT, as stated in the Sage-S2.1-Flash-FP8 model card by Shanghai AI Laboratory.
- The WEX format, codecs, specifications and tools (not stored here): Apache-2.0.
The license texts are in licenses/, copied unchanged from the upstream repositories.
Follow the license of each component.