A 284B mixture, on a board you can hold

DeepSeek V4 Flash (0731), MaxQ-Fractal 64 GB β€” the ARM64 + CUDA build, the kernel fix that made it correct, and the measurements, including the two we retracted.

This repository does not contain the weights. The checkpoint is Synchopate/DeepSeek-V4-Flash-0731-MaxQ-Fractal-64GB-CPU-GGUF β€” MIT, 284.33 B total / 13.17 B active across three shards, 56.93 GiB. What is here is everything that was missing if you wanted to run it anywhere other than an x86 desktop.

upstream this repo
Weights yes, 3 shards no β€” pinned by SHA-256
Binaries x86_64 (generic, AVX2, AVX512) aarch64, CPU + CUDA sm_87
CUDA q2_0 decoder reads the old block layout, so it decodes ~zero fixed, verified bit-exact
Conformance gate none published golden vectors, signed report
Measured on a named box CPU desktop Jetson AGX Orin 64 GB, flags and date stated
Retractions none two numbers withdrawn, shown with the reason

Measured, on one box, with the flags in the command

Jetson AGX Orin 64 GB (MAXN), 12 ARM cores, 204.8 GB/s. 2026-11-15.

value how
decode 2.06 Β± 0.16 tok/s llama-bench -ngl 28 -p 0 -n 32 -t 12, no HTTP server in the loop; the server measures 2.08 at -ngl 34, so the engine is the ceiling, not the plumbing
prefill, 1,076-token prompt 37.98 tok/s same record, -ngl 34 -fa on -ctk q8_0 -ctv q8_0
resident 56.93 GiB checkpoint on disk
load 31–90 s 57 GiB of shards
output coherent verified against the CPU decoder

Two numbers elsewhere in the wild for this checkpoint, withdrawn: 70.7 tok/s prefill and 9.52 tok/s decode. They were measured while the CUDA decoder was producing zeros; zeroed attention also flattened MoE routing, which is why those runs looked fast. The honest comparison to the CPU build is 25Γ— prefill, 1.5Γ— decode. We would rather show you the wrong numbers with the reason than have you find them without it.

The bug, because it is the useful part

llama-modchip redefines block_q2_0 in place β€” a 30-byte block, 64 weights, 3-bit codebook, eight 4-bit sub-block scales β€” and reuses the deprecated enum slot 42. The fork's CPU decoder was updated for the new layout. Its CUDA decoder was not: ggml/src/ggml-cuda/dequantize.cuh still read the old 32-weight, 18-byte block. Every weight it decoded was approximately zero.

That is why the failure looked like a model problem and was a one-file problem. The fix is four edits β€” the decoder ported from the CPU reference, and three dispatch gates that keep the stale fast kernels away from the type β€” applied by scripts/patch_dsv4_cuda_q2_0.py, which is idempotent and refuses a drifted tree rather than writing half a patch.

Verified before it was trusted: the new index arithmetic reproduces the CPU reference on 19,200 weights over 300 blocks β€” 0 mismatches, and the patched kernel compiles clean for sm_87. On the GPU path, the type is expanded through f16 per call, which is correct but not fast; that cost is measured and stated, not hidden.

Quick start

git clone https://github.com/NGARiAI/ns-bos-kernel        # the kernel, optional but it is what makes the receipts
git lfs install
huggingface-cli download Synchopate/DeepSeek-V4-Flash-0731-MaxQ-Fractal-64GB-CPU-GGUF \
  --local-dir ./dsv4

# the aarch64 build in this repo
LD_LIBRARY_PATH=./aarch64 ./aarch64/llama-server \
  -m ./dsv4/DeepSeek-V4-Flash-0731-MaxQ-Fractal-00001-of-00003.gguf \
  -ngl 20 -c 2048 -t 12 --host 127.0.0.1 --port 8080

-c 2048 is not a typo. On a 64 GB board the weights leave about 4 GB, so the context is the thing you budget. If you have 128 GB, try 8192 and tell us what happens β€” see below.

Reproduce the numbers

python3 scripts/dsv4_bench.py --launch --unload-ollama
python3 scripts/q2_0_conformance.py --check      # exit 2 means it could not test, not that it passed

The conformance suite is the point: golden vectors that each isolate one rule of the format β€” the sub-block scale, the high-bit plane, the codebook index, the extremes β€” plus a deterministic pseudo-random set, with a signed report naming the vectors' hash and the box. A format that reports numbers before it can reproduce fixed vectors is asking to be believed.

Licences, in full

what licence note
Weights, MaxQ-Fractal checkpoint MIT Synchopate's quant of deepseek-ai/DeepSeek-V4-Flash-0731 (MIT)
llama-modchip engine and binaries Apache-2.0 synchopate/llama-modchip; NOTICE retained
The CUDA fix, applier, conformance suite, instrument MIT this repository

If you redistribute the binaries, ship the NOTICE. If you redistribute the weights, MIT asks only for the notice β€” but the checkpoint will not load in stock llama.cpp, ollama, vLLM or LM Studio. It needs GGML_TYPE_Q2_0 = 42 and the deepseek4 architecture, which is this fork.

What is not measured, and we are not going to imply otherwise

  • Quality. No perplexity, no accuracy suite, no benchmark score for this checkpoint has been run by anyone we can find, and we have not run one either. It is the largest hole in our own doctrine and it is named as a hole, in public. A number invented here would be worth less than the absence.
  • Speed on anything but this board. One Orin, one set of flags, one date.
  • Long context. 2,048 tokens on 64 GB. The model card claims 1,048,576; that is a card claim, not a run.

If you have a second box

We have exactly one of these running, and one measurement is not a benchmark. If you run it on different hardware β€” an Orin AGX, a DGX Spark, Threadripper, Strix Halo, a rented H100 β€” run dsv4_bench.py and open an issue with the JSON. We publish the row with your box's name on it, whether it flatters ours or not. The interesting ones are the 128 GB unified-memory machines: they may hold 8k–32k of context where we hold 2k, and nobody has measured it.


NGARi builds sovereign AI: models that run on hardware you own, with a receipt for every byte that leaves the box. This checkpoint is the extreme of that claim β€” a 284B mixture at 1.72 bits per weight, on a board that fits in a bag. ngari.ai Β· kernel Β· the doctrine

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for NGARiAI/DeepSeek-V4-Flash-284B-Edge-ARM64