DeepSeek-V4.1-Flash-1017 — REAP-192 (thor-v1 Pure-CUDA Build)
-brightgreen){kind=icon}
106.9 GiB pure-CUDA thor-v1 build of 50%-pruned DeepSeek-V4.1-Flash-1017 (192 of 384 routed experts) for a single 128 GB NVIDIA Jetson AGX Thor (sm_110a) with O_DIRECT NVMe Engram streaming.
| Sibling Artifact | Format | Size | Target Hardware / Runtime |
|---|---|---|---|
DeepSeek-V4.1-Flash-1017-REAP192-MXFP4 |
Native MXFP4 + FP8 (safetensors) |
146.4 GiB | Bit-exact 192-expert master checkpoint |
DeepSeek-V4.1-Flash-1017-REAP192-Thor (this repo) |
thor-v1 (2.828 bpw K-quant + FP8/BF16) |
106.9 GiB | Single 128 GB NVIDIA Jetson AGX Thor (v41_server, 8.09 tok/s, 1M ctx) |
DeepSeek-V4.1-Flash-1017-REAP192-GGUF |
3-shard GGUF L3 (2.46 bpw I-quant + Q8_0/Q6_K) |
85.55 GiB | llama.cpp (PR #28696 + 6 patches, -ngl 999, 11.86 tok/s on 128 GB Thor) |
patrickbdevaney/deepseek-v4.1-flash-reap-thor |
Pipeline, cuda/ server & llama.cpp patches |
— | Full build pipeline (ops/s01..s12), cuda/, and gates |
DeepSeek-V4.1-Flash-1017-REAP192-Thor is deepseek-ai/DeepSeek-V4.1-Flash-1017 pruned to 192 of 384 routed experts (keep-set 1ace9b33134a2ca7, recorded in reap_keepset.json), in the thor-v1 layout (thor.json, cuda/FORMAT_PROPOSAL.md). That layout is built for one 128 GB NVIDIA Jetson Thor running a pure-CUDA decoder (cuda/), with the Engram tables read from NVMe rather than held in memory.
Text only for now. The vision tower and aligner (vision.safetensors) are in this build unchanged, and the pruning protected the experts image tokens route to, but the CUDA server here does not run the vision tower yet: image inputs are not accepted. Vision support is planned as a server update to this repository; the weights will not change.
✨ Highlights
- Single-Box 128 GB Jetson AGX Thor Fit (
114.9 / 117.0 GiB): Resident footprint is 106.9 GiB (89.5 GiBrouted experts at2.828 bpwacrosslayer-00.safetensors..layer-39.safetensors+ native FP8/BF16 dense MLA/DSA/shared-expert/router/mHC path indense.safetensors), leaving budget for the 2.0 GiB 1M-token context state (FP8 KV + indexer keys) and 6.0 GiB decoder workspace. - 8.09 tok/s Pure-CUDA Decode with O_DIRECT NVMe Engram Streaming: Achieves 8.09 tok/s at short context and 7.14 tok/s after an 8k-token prompt (greedy, batch 1) while streaming Engram rows per token from local NVMe via
O_DIRECT(ops/s09_engram_rowfile.py). - GPTQ Error-Feedback K-Quant Experts with Folded Healing Gain: Routed experts were requantized once from the MXFP4 source with GPTQ error-feedback rounding (
ops/s07_quant.py) into ggml K-quant blocks (w13:Q2_K 31, w13:Q3_K 9, w2:Q2_K 28, w2:Q3_K 12), with the per-layer healing gain folded intow2before rounding. - Verified Teacher-Forced Fidelity: Achieves 96.87% top-1 agreement (
kl32 = 0.0607) on code, 92.94% (kl32 = 0.1509) on math, 86.46% (kl32 = 0.3663) on vision, and 84.56% (kl32 = 0.2265) on agentic tasks (0.7756overall across 185,565 confirmation tokens), outperforming the bit-exact 128-expert alternative (top-1 = 0.7092,kl32 = 0.8026) by a wide margin.
📋 Model Summary
| Property | Value |
|---|---|
| Base Model | deepseek-ai/DeepSeek-V4.1-Flash-1017 (1bae0e1088993b9b82de08f59de5be5a34e42170, config.json) |
| Master Checkpoint | patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-MXFP4 |
| Layout / Format | thor-v1 (thor.json manifest with SHA256 hashes for every file) |
| Routed Experts | 192 / 384 per layer (reap_keepset.json, 1ace9b33134a2ca7), 2.828 bpw (w13:Q2_K 31, Q3_K 9; w2:Q2_K 28, Q3_K 12), 89.5 GiB |
| Dense Path | Native base checkpoint FP8/BF16 tensors (dense.safetensors, mtp-0.safetensors..mtp-2.safetensors, vision.safetensors) |
| Total Build Size | 106.9 GiB resident (114.9 GiB total with 2.0 GiB 1M-ctx state + 6.0 GiB workspace on 128 GB Jetson Thor) |
| Measured Decode Speed | 8.09 tok/s (short context) / 7.14 tok/s (after 8k-token prompt), greedy batch=1, Engram on NVMe |
| Runtime | v41_server (cuda/README.md) |
| License | MIT (LICENSE) |
Format
Experts were requantized once from the MXFP4 source with GPTQ error-feedback rounding into ggml K-quant blocks (ops/s07_quant.py; per-layer mix, protected layers higher; counts of layer x projection-group: w13:Q2_K 31, w13:Q3_K 9, w2:Q2_K 28, w2:Q3_K 12). The heal gain was folded into w2 before rounding.
- Experts: 2.828 bits per weight on average, 89.5 GiB in total (
layer-00.safetensors..layer-39.safetensors). - Whole build: 106.9 GiB (
thor.jsonlists every file with its sha256). The dense path (dense.safetensors: attention, indexer, compressor, shared expert, router, hyper-connections) is the base checkpoint's own FP8/BF16 tensors. The vision tower and aligner (vision.safetensors) are unpruned. The checkpoint's own speculative-decoding (mtp) blocks (mtp-0.safetensors,mtp-1.safetensors,mtp-2.safetensors) are included unmodified. - Keep-set:
reap_keepset.jsonlists, per layer, the routed experts that were kept, in their original ids. The router rows and biases were sliced to the same ids.
Memory on the Thor
| item | GiB |
|---|---|
| this build, resident | 106.9 |
| main 1M-token context state (FP8 KV + indexer keys) | 2.0 |
| decoder workspace | 6.0 |
| total | 114.9 of a 117.0 GiB budget: fits |
The context-state figure is the pre-registered default (not measured yet). The Engram tables are not in memory: they are read per token from row files on NVMe (below).
Decode speed: 8.09 tok/s at a short context, 7.14 tok/s after an 8k-token prompt (greedy, batch 1, Engram rows on NVMe; measured by the s08 decode gate on this build).
How it was evaluated
Every number below comes from s10/final/results.json of the build pipeline (ops/s10_eval.py, RESULTS.md). It is a paired, teacher-forced comparison against the unpruned deepseek-ai/DeepSeek-V4.1-Flash-1017 at revision 1bae0e1088993b9b82de08f59de5be5a34e42170, run in its own MXFP4/FP8 numerics. The comparison uses 185565 scored tokens from 26 held-out documents in the confirmation split. That split was never used to choose the model. The intervals are ninety-five percent document-cluster bootstrap intervals.
- kl32 is the mean over tokens of the KL divergence on the teacher's top-32 support.
- top-1 agreement is the share of tokens where this model's argmax equals the teacher's.
- dNLL is this model's next-token NLL minus the teacher's.
The pass/fail thresholds were pre-registered before any result existed (ops/s10_gates.json, sha256 f95b26721124c6a3). Any threshold amended after a result is listed below with its date, and the result under the original rule is shown beside it (DECISIONS.md).
| arm | kl32 (95% CI) | top-1 agreement | dNLL | confident-flip rate |
|---|---|---|---|---|
| control | 0.0000 [0.0000, 0.0000] | 1.0000 [1.0000, 1.0000] | 0.0000 [0.0000, 0.0000] | 0.0000 |
| mxfp4_A | 0.4405 [0.2639, 0.6465] | 0.7980 [0.7431, 0.8481] | 0.2718 [0.1219, 0.4585] | 0.0524 |
| thorA | 0.5194 [0.3261, 0.7423] | 0.7756 [0.7179, 0.8284] | 0.3091 [0.1426, 0.5119] | 0.0619 |
| b128 | 0.8026 [0.5627, 1.0801] | 0.7092 [0.6443, 0.7701] | 0.5469 [0.3271, 0.8010] | 0.0995 |
Known limitation: multilingual. Top-1 agreement with the unpruned model on the held-out multilingual documents is lower than in every other domain (confirmation split: mxfp4_A 0.5247, thorA 0.4935, b128 0.4219). The release floor for every other domain is 0.65; for multilingual it was lowered to a collapse guard of 0.4 (2026-10-22, owner decision, before any s10 result). During expert selection, the shipped keep-set scored 0.5233 on the screen split against 0.5592-0.5741 for random keep-sets of the same size. Keep-sets that protected more multilingual experts recovered multilingual only by losing general, finance and science text, so the better all-round keep-set ships. For multilingual use, prefer the unpruned model.
Gate amended after the result (2026-10-24). The pre-registered rule required the pruned model to beat a random keep-set of the same size on kl32. Over all domains it did not reach significance (paired difference -0.1014, interval -0.2196 to +0.0400), so this release FAILED its gate as originally registered. The cause is multilingual alone. After seeing that result, the owner amended the rule to exclude multilingual, whose weakness is already documented above. Without it the difference is -0.2144 (interval -0.2633 to -0.1676). Reason: Owner decision AFTER the first s10 final result (confirm split). As registered, mxfp4_A vs random0 paired d_kl32 was -0.1014 [-0.2196, +0.0400]: not significant, entirely because of the multilingual bucket (kl32 1.534 vs random 0.894), the weakness the owner had already accepted on 2026-10-22 with a lowered top-1 floor. In every other bucket the keep-set has roughly half of random's kl32. The as-registered comparison stays in results.json and on the cards beside the amended one; no other threshold changes (thorA top-1 0.78 floor still applies).
Thor build: top-1 floor waived (2026-10-24, after the result). The low-bit Thor build scored top-1 0.7756 (interval 0.7179 to 0.8284) against a pre-registered floor of 0.78, so it FAILED that floor; the MXFP4 master scores 0.7980. The owner waived this one clause for the Thor build. Every other clause still applies and passed. Reason: Owner decision after the s10 final result. thorA top-1 0.7756 [0.7179, 0.8283] misses the 0.78 sanity floor (borrowed from GLM REAP50 v2's 0.842) by 0.0044, well inside its interval. It is the only 192-expert build that fits Thor; the bit-exact 128-expert alternative is far worse (top-1 0.7092, paired kl32 +0.283 vs thorA). Requantization costs kl32 +0.079 vs the MXFP4 master (limit 0.25) and top-1 -0.022. Every other clause still applies.
control keeps all experts and is re-run through the same evaluation machinery as the others. It measures that machinery, not the model.
This build is the thorA arm (numbers from s10/final/results.json):
| arm | kl32 (95% CI) | top-1 agreement | dNLL | confident-flip rate |
|---|---|---|---|---|
| thorA | 0.5194 [0.3261, 0.7423] | 0.7756 [0.7179, 0.8284] | 0.3091 [0.1426, 0.5119] | 0.0619 |
By domain:
| domain | tokens | kl32 | top-1 agreement | dNLL |
|---|---|---|---|---|
| agentic | 40955 | 0.2265 [0.1998, 0.2526] | 0.8456 [0.8369, 0.8558] | 0.0916 [0.0623, 0.1168] |
| code | 24573 | 0.0607 [0.0403, 0.0937] | 0.9687 [0.9569, 0.9759] | 0.0439 [0.0277, 0.0698] |
| finance | 24573 | 0.4582 [0.2725, 0.7006] | 0.7440 [0.6859, 0.7810] | 0.2803 [0.1667, 0.4332] |
| general | 32764 | 0.5545 [0.4610, 0.6687] | 0.7523 [0.7078, 0.7882] | 0.1475 [-0.1085, 0.4034] |
| math | 8191 | 0.1509 [0.1509, 0.1509] | 0.9294 [0.9294, 0.9294] | 0.0894 [0.0894, 0.0894] |
| multilingual | 24573 | 1.7097 [1.4440, 1.9126] | 0.4935 [0.4415, 0.5412] | 1.3697 [1.0515, 1.6590] |
| science | 24573 | 0.4466 [0.3384, 0.6436] | 0.7401 [0.7089, 0.7707] | 0.3136 [0.2241, 0.4637] |
| vision | 5363 | 0.3663 [0.3278, 0.4002] | 0.8646 [0.8556, 0.8723] | -0.2403 [-0.3219, -0.1661] |
CUDA correctness gates (layer by layer against the torch reference, cuda/tests/gate_v41_golden.cu). Source checkpoint: 33 of 41 teacher-forced next-token disagreements were near-ties (80.5%), FAIL as registered (rule: at least 90%); passed under an owner amendment made before its diagnostic was read, for chained bf16 drift (ops/decisions/2026-10-24-golden-src-drift-amendment.md). This build: 41 of 46 teacher-forced next-token disagreements were near-ties (89.1%), FAIL as registered (rule: at least 90%); the owner waived it and released the build on the serving gates (Engram row store, decode determinism, 1M context, server) (ops/decisions/2026-10-25-golden-a192-drift-amendment.md, which also records the per-row diagnostics). Every hidden-state layer check and the agreement and final-cosine rules passed in both runs.
Release gate: PASS under the gate as amended 2026-10-24 (FAIL as originally registered; see the evaluation). The repository is public only if the pre-registered gate for this arm passed and the build fits the budget above.
How to run it
The CUDA server that runs this format (v41_server, cuda/) is in the companion pipeline repository:
- Download this repository and the two Engram shards of the base model (below).
- Build the Engram row files from those two shards (
ops/s09_engram_rowfile.pybuild, thenverify --full). The row files are about the size of the two shards and must sit on a local NVMe drive: they are read with O_DIRECT for every token. - Start the server:
v41_server --ckpt <this repo> --keepset <this repo>/reap_keepset.json --engram <row-file dir> --host 127.0.0.1 --port 18108
It serves an OpenAI-style /v1/chat/completions (streaming and tool calls, with the model's own chat encoding in encoding/encoding.py) and /health.
Speculative decoding
Not measured yet. The checkpoint's own DSpark head (the three mtp layers) is being refit to this keep-set; this section is filled from that result.
Engram tables (required, not included)
model-00047-of-00048.safetensors(101.5 GB, sha25676a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5)model-00048-of-00048.safetensors(101.5 GB, sha2569dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd)
hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 model-00047-of-00048.safetensors model-00048-of-00048.safetensors --revision 1bae0e1088993b9b82de08f59de5be5a34e42170 --local-dir .
🔎 License & Provenance Ledger
| Claim / Artifact | Primary Verification Source | Digest / Revision |
|---|---|---|
| License | LICENSE (MIT, copied verbatim from base model) |
MIT |
| Upstream Teacher | deepseek-ai/DeepSeek-V4.1-Flash-1017 |
1bae0e1088993b9b82de08f59de5be5a34e42170 |
| Master Pruned Checkpoint | patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-MXFP4 |
Keep-set 1ace9b33134a2ca7.. |
| Per-Shard Manifest & Checksums | thor.json & reap_keepset.json (in this repo) |
thor-v1 (106.9 GiB) |
| GPTQ Error-Feedback Quantizer | ops/s07_quant.py & cuda/FORMAT_PROPOSAL.md |
2.828 bpw routed experts |
| CUDA Golden Correctness Gates | cuda/tests/gate_v41_golden.cu, ops/decisions/2026-10-24-golden-src-drift-amendment.md, ops/decisions/2026-10-25-golden-a192-drift-amendment.md |
Commit 36bd04c |
| Pre-Registered Gates & Eval Results | ops/s10_gates.json, DECISIONS.md, RESULTS.md |
f95b26721124c6a343f41b004b8740bfe6eb22033d77dcc260dceda039082e0a |
Model tree for patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-Thor
Base model
deepseek-ai/DeepSeek-V4.1-Flash-1017