DeepSeek-V4-Flash MTP layer converted for MLX

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 DeepSeek
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,73 @@
+---
+license: mit
+library_name: mlx
+base_model: deepseek-ai/DeepSeek-V4-Flash
+base_model_relation: quantized
+tags:
+- mlx
+- deepseek_v4
+- speculative-decoding
+- tensorfold
+---
+
+# DeepSeek-V4-Flash MTP for MLX
+
+This is DeepSeek-V4-Flash's own multi-token-prediction (MTP) layer, converted for Apple Silicon. The mlx-community
+4-bit conversion leaves it out. It drafts tokens for
+[mlx-community/DeepSeek-V4-Flash-4bit](https://huggingface.co/mlx-community/DeepSeek-V4-Flash-4bit) on
+[TensorFold](https://github.com/ashhart/TensorFold)'s lane engine, which verifies every draft against the target.
+
+- Source: [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) at revision
+  `60d8d70770c6776ff598c94bb586a859a38244f1`. The layer is the `mtp.0.*` tensors of
+  `model-00046-of-00046.safetensors`.
+- License: MIT, DeepSeek's (see `LICENSE`, copyright 2023 DeepSeek). The conversion adds no terms.
+- Files: `model.safetensors` (3.52 GB), `config.json`, `LICENSE`, `SHA256SUMS`.
+
+## Use
+
+TensorFold 0.3.6.4 or later on a Mac with 256 GB:
+
+```bash
+tensorfold pull mlx-community/DeepSeek-V4-Flash-4bit Vontra/DeepSeek-V4-Flash-MTP-MLX
+tensorfold serve mlx-community/DeepSeek-V4-Flash-4bit --drafter Vontra/DeepSeek-V4-Flash-MTP-MLX
+```
+
+The layer drafts up to three tokens a round, one pass each, from the target's streams at the last verified row. A
+drafted reply equals the same request sent with `"draft": false`. Drafts change speed, never tokens. TensorFold's
+default draft head is DeepSeek's DSpark module,
+[Vontra/DeepSeek-V4-Flash-DSpark-MLX](https://huggingface.co/Vontra/DeepSeek-V4-Flash-DSpark-MLX) (10.67 GB).
+
+## What the conversion does
+
+- The layer's dense projections, `e_proj` and `h_proj` included, are FP8 (E4M3) with an E8M0 scale per 128 x 128
+  block. They are dequantized to bf16, which is exact, then quantized to MLX affine 4-bit in groups of 64. That step
+  is lossy, and it is the format the mlx-community target stores for its own dense projections.
+- The routed experts keep DeepSeek's mxfp4 bytes and E8M0 scales unchanged.
+- Norms (`enorm`, `hnorm` and the block's own), attention sinks, router weights and biases and hyper-connection
+  tensors are copied as stored.
+- Block `mtp.0` becomes `mtp`, and its tensors take the names the mlx-community target uses for its own layers
+  (`attn.wq_a`, `ffn.switch_mlp.gate_proj` and so on). `config.json` names the head with
+  `"model_type": "deepseek_v4_mtp"`.
+
+TensorFold's converter reproduces `model.safetensors` byte for byte from the source:
+
+```bash
+python -m tensorfold.families.deepseek_v4.convert mtp model-00046-of-00046.safetensors DeepSeek-V4-Flash-MTP-MLX
+```
+
+## Measured
+
+On an M3 Ultra (60-core GPU, 256 GB) with MLX 0.32.2, TensorFold's lane engine served four short chat prompts (a
+Fibonacci function, how a GPU runs matrix multiplication, primes, a translation), 64 generated tokens each. Decode
+tok/s with this drafter and with `"draft": false`; sampled is temperature 1.0 with seed 1234:
+
+| Prompt | Greedy, drafted | Greedy, no drafts | Sampled, drafted | Sampled, no drafts |
+| --- | --- | --- | --- | --- |
+| 1 | 60.9 | 46.5 | 53.6 | 45.8 |
+| 2 | 53.0 | 46.4 | 48.1 | 45.7 |
+| 3 | 64.2 | 46.6 | 59.5 | 45.8 |
+| 4 | 58.7 | 46.4 | 60.7 | 45.7 |
+
+Each drafted reply equals its `"draft": false` reply. For comparison, mlx-lm PR #1797's server decodes 23.0-23.5
+tok/s greedy and 28.7-29.4 sampled on the same machine and weights with its defaults, measured on a chat and code
+fixture rather than these four prompts.

SHA256SUMS

@@ -0,0 +1,4 @@
+f2c6c602815669d292889e5be8c802f2ed950653b77999b1584e8e6aed25d040  LICENSE
+bccbcaaa23b9ecd802f526cd9ba0b0b44736385084f253d09e43f822a82df650  README.md
+b760dd370e395fc731c9dc0b484f7f5aa2a1221cfc21e489280f712512d3cb3e  config.json
+86238539c7c0b4a4f1916806fc2eb280843413a52e91fac7cb8e2ed32628b89e  model.safetensors

config.json

@@ -0,0 +1,3 @@
+{
+ "model_type": "deepseek_v4_mtp"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86238539c7c0b4a4f1916806fc2eb280843413a52e91fac7cb8e2ed32628b89e
+size 3521300333