A 284B mixture, on a board you can hold
DeepSeek V4 Flash (0731), MaxQ-Fractal 64 GB β the ARM64 + CUDA build, the kernel fix that made it correct, and the measurements, including the two we retracted.
This repository does not contain the weights. The checkpoint is
Synchopate/DeepSeek-V4-Flash-0731-MaxQ-Fractal-64GB-CPU-GGUF
β MIT, 284.33 B total / 13.17 B active across three shards, 56.93 GiB. What is here is everything that
was missing if you wanted to run it anywhere other than an x86 desktop.
| upstream | this repo | |
|---|---|---|
| Weights | yes, 3 shards | no β pinned by SHA-256 |
| Binaries | x86_64 (generic, AVX2, AVX512) | aarch64, CPU + CUDA sm_87 |
CUDA q2_0 decoder |
reads the old block layout, so it decodes ~zero | fixed, verified bit-exact |
| Conformance gate | none published | golden vectors, signed report |
| Measured on a named box | CPU desktop | Jetson AGX Orin 64 GB, flags and date stated |
| Retractions | none | two numbers withdrawn, shown with the reason |
Measured, on one box, with the flags in the command
Jetson AGX Orin 64 GB (MAXN), 12 ARM cores, 204.8 GB/s. 2026-11-15.
| value | how | |
|---|---|---|
| decode | 2.06 Β± 0.16 tok/s | llama-bench -ngl 28 -p 0 -n 32 -t 12, no HTTP server in the loop; the server measures 2.08 at -ngl 34, so the engine is the ceiling, not the plumbing |
| prefill, 1,076-token prompt | 37.98 tok/s | same record, -ngl 34 -fa on -ctk q8_0 -ctv q8_0 |
| resident | 56.93 GiB | checkpoint on disk |
| load | 31β90 s | 57 GiB of shards |
| output | coherent | verified against the CPU decoder |
Two numbers elsewhere in the wild for this checkpoint, withdrawn: 70.7 tok/s prefill and 9.52 tok/s decode. They were measured while the CUDA decoder was producing zeros; zeroed attention also flattened MoE routing, which is why those runs looked fast. The honest comparison to the CPU build is 25Γ prefill, 1.5Γ decode. We would rather show you the wrong numbers with the reason than have you find them without it.
The bug, because it is the useful part
llama-modchip redefines block_q2_0 in place β a 30-byte block, 64 weights, 3-bit codebook, eight
4-bit sub-block scales β and reuses the deprecated enum slot 42. The fork's CPU decoder was
updated for the new layout. Its CUDA decoder was not: ggml/src/ggml-cuda/dequantize.cuh still read
the old 32-weight, 18-byte block. Every weight it decoded was approximately zero.
That is why the failure looked like a model problem and was a one-file problem. The fix is four edits β
the decoder ported from the CPU reference, and three dispatch gates that keep the stale fast kernels
away from the type β applied by scripts/patch_dsv4_cuda_q2_0.py, which is idempotent and refuses a
drifted tree rather than writing half a patch.
Verified before it was trusted: the new index arithmetic reproduces the CPU reference on 19,200 weights
over 300 blocks β 0 mismatches, and the patched kernel compiles clean for sm_87. On the GPU path, the
type is expanded through f16 per call, which is correct but not fast; that cost is measured and stated,
not hidden.
Quick start
git clone https://github.com/NGARiAI/ns-bos-kernel # the kernel, optional but it is what makes the receipts
git lfs install
huggingface-cli download Synchopate/DeepSeek-V4-Flash-0731-MaxQ-Fractal-64GB-CPU-GGUF \
--local-dir ./dsv4
# the aarch64 build in this repo
LD_LIBRARY_PATH=./aarch64 ./aarch64/llama-server \
-m ./dsv4/DeepSeek-V4-Flash-0731-MaxQ-Fractal-00001-of-00003.gguf \
-ngl 20 -c 2048 -t 12 --host 127.0.0.1 --port 8080
-c 2048 is not a typo. On a 64 GB board the weights leave about 4 GB, so the context is the thing you
budget. If you have 128 GB, try 8192 and tell us what happens β see below.
Reproduce the numbers
python3 scripts/dsv4_bench.py --launch --unload-ollama
python3 scripts/q2_0_conformance.py --check # exit 2 means it could not test, not that it passed
The conformance suite is the point: golden vectors that each isolate one rule of the format β the sub-block scale, the high-bit plane, the codebook index, the extremes β plus a deterministic pseudo-random set, with a signed report naming the vectors' hash and the box. A format that reports numbers before it can reproduce fixed vectors is asking to be believed.
Licences, in full
| what | licence | note |
|---|---|---|
| Weights, MaxQ-Fractal checkpoint | MIT | Synchopate's quant of deepseek-ai/DeepSeek-V4-Flash-0731 (MIT) |
llama-modchip engine and binaries |
Apache-2.0 | synchopate/llama-modchip; NOTICE retained |
| The CUDA fix, applier, conformance suite, instrument | MIT | this repository |
If you redistribute the binaries, ship the NOTICE. If you redistribute the weights, MIT asks only for
the notice β but the checkpoint will not load in stock llama.cpp, ollama, vLLM or LM Studio.
It needs GGML_TYPE_Q2_0 = 42 and the deepseek4 architecture, which is this fork.
What is not measured, and we are not going to imply otherwise
- Quality. No perplexity, no accuracy suite, no benchmark score for this checkpoint has been run by anyone we can find, and we have not run one either. It is the largest hole in our own doctrine and it is named as a hole, in public. A number invented here would be worth less than the absence.
- Speed on anything but this board. One Orin, one set of flags, one date.
- Long context. 2,048 tokens on 64 GB. The model card claims 1,048,576; that is a card claim, not a run.
If you have a second box
We have exactly one of these running, and one measurement is not a benchmark. If you run it on
different hardware β an Orin AGX, a DGX Spark, Threadripper, Strix Halo, a rented H100 β run
dsv4_bench.py and open an issue with the JSON. We publish the row with your box's name on it, whether
it flatters ours or not. The interesting ones are the 128 GB unified-memory machines: they may hold
8kβ32k of context where we hold 2k, and nobody has measured it.
NGARi builds sovereign AI: models that run on hardware you own, with a receipt for every byte that leaves the box. This checkpoint is the extreme of that claim β a 284B mixture at 1.72 bits per weight, on a board that fits in a bag. ngari.ai Β· kernel Β· the doctrine