Upload folder using huggingface_hub

.meta.json

@@ -0,0 +1,11 @@
+{
+  "Pillow": "12.3.0",
+  "flashinfer-python": "0.6.15.post1",
+  "huggingface_hub": "1.27.0",
+  "numpy": "2.5.2",
+  "safetensors": "0.8.0",
+  "sympy": "1.14.0",
+  "tilelang": "0.1.12",
+  "torch": "2.13.0+cu126",
+  "transformers": "5.15.0.dev0"
+}

README.md

@@ -0,0 +1,595 @@
+---
+library_name: transformers
+base_model:
+- deepseek-ai/DeepSeek-V4.1-Flash
+---
+
+This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash).
+
+Note:
+  - Structural comparison with the original `deepseek-ai/DeepSeek-V4.1-Flash`:
+
+| Structural feature | Original V4.1 Flash | This tiny model |
+|---|---|---|
+| Modalities | Vision + text | Same |
+| Vocabulary / special-token IDs | 129,280; original BOS/EOS/PAD/image IDs | Same |
+| Text backbone layers | 40 | 6 |
+| Hidden size | 5,120 | 128 |
+| Attention heads / head dim | 64 / 512 | 4 / 512 (SGLang DSV4 ABI; `head_dim >= 32`) |
+| RoPE head dim | 64 | Same |
+| Q/O low-rank projections | q=1,280; o=1,024; 8 output groups | q=128; o=128; 2 groups |
+| Routed/shared experts | 384 / 1, top-6 | 32 / 1, top-6 |
+| MoE intermediate size | 2,304 | 256 (128 per TP rank) |
+| Sparse attention ratios | Backbone: 2 sliding-only, 18 ratio-2, 20 ratio-1; MTP: 3 sliding-only | Backbone: 2 sliding-only, 2 ratio-2, 2 ratio-1; MTP: 3 sliding-only |
+| KV / index source layers | KV: 2,8,14,20; index: 2,8,14,20,24,28,32,36 | KV/index: 1,3; index head dim 128 |
+| Candidate source layer | 20 | 3 |
+| Hyper-Connections | 4 residual streams | Same |
+| Engram | layers 1,14; 8 heads x 256; ~384M rows/table | layers 1,2; 4 heads x 64; 6,748 / 7,592 rows |
+| DSpark/MTP | 3 stages; targets 37,38,39; Markov rank 256; 128 experts, top-3 | 3 stages; targets 3,4,5; rank 16; 8 experts, top-3 |
+| Vision tower | 32 layers; width 1,024; 16 heads | 2 layers; width 64; 4 heads |
+| Vision patch/downsample | patch 14; downsample 3 | Same |
+| Quantization | Routed experts FP4; most other linears FP8; UE8M0 scales; 32x32 blocks | Same |
+| Public checkpoint | 96,085 tensors; ~510.3 GB | 1,629 tensors; 96.7 MB |
+
+  - Preserved topology includes compressed sparse attention and its indexer, four-way
+    Hyper-Connections, n-gram Engram lookup, three-stage DSpark/MTP, the vision aligner,
+    FP4 expert packing, and FP8 block scales. Widths, layer cadence, expert counts,
+    Engram tables, and vision depth are reduced for a practical debugging checkpoint.
+
+  - The structure is derived by instantiating DeepSeek's pinned official V4.1 inference
+    implementation on the meta device, then converting its runtime state back to the
+    public Hugging Face checkpoint representation. The original and 1017 releases have
+    byte-identical source `config.json` and checkpoint index (96,085 tensor names), so
+    the 1017 source tree is also the authoritative executable structure for this release.
+
+  - The SGLang example is intentionally a conservative TP=2 launch. The
+    attention latent stays 512-wide and the indexer stays 128-wide because those are
+    fixed by SGLang's DSV4 cache/kernel ABI; all TP-sharded widths, head/group counts,
+    expert counts, vocabulary rows, and 32-wide quantization blocks divide cleanly by 2.
+    The online example enables the embedded three-stage DSpark/MTP draft explicitly.
+
+  - Validated on two NVIDIA H20 GPUs with Python 3.12.13, CUDA Toolkit 13.0,
+    torch 2.14.1, SGLang `0.5.23.dev384+g9ab74b851`, FlashInfer 0.7.2,
+    Transformers 5.19.0, TileLang 0.1.12, and safetensors 0.8.0. Separate offline
+    `sglang.Engine` runs using the same TP=2/backend settings reached engine-ready
+    with both base decoding and `speculative_algorithm="DSPARK"`; CUDA Graph remained
+    enabled, and the latter reported DSpark enabled, generated all three cases
+    (including `Hello` with exactly 10 new tokens), and shut down with exit code 0.
+    This artifact is byte-identical to the
+    separately generated 1017 tiny variant used for that runtime test (SHA-256
+    `53aae999fdcacd310c400a2fc3112bb68ce35588441784c75dca5820abf0f35b`). The README
+    keeps the equivalent online `sglang serve` form for normal deployment. The
+    official source `inference/convert.py` also produced two shards which loaded
+    strictly into the source runtime for both ranks.
+
+| File path | Size |
+|------|------|
+| model.safetensors | 96.7MB |
+
+
+### Example usage:
+
+- SGLang
+
+```bash
+# Requires an SGLang preview/main build with DeepSeek-V4.1 support.
+# Runs the target model and its embedded three-stage DSpark/MTP draft with TP=2.
+model_id=tiny-random/deepseek-v4.1-flash
+sglang serve \
+  --model-path $model_id \
+  --trust-remote-code \
+  --tp-size 2 \
+  --ep-size 1 \
+  --attention-backend dsv4 \
+  --moe-runner-backend flashinfer_mxfp4 \
+  --speculative-algorithm DSPARK \
+  --speculative-dspark-block-size 5 \
+  --context-length 4096 \
+  --max-total-tokens 4096 \
+  --chunked-prefill-size 512 \
+  --mem-fraction-static 0.5 \
+  --disable-custom-all-reduce
+```
+
+### Codes to create this repo:
+
+<details><summary>Click to expand</summary>
+
+```python
+import json
+import hashlib
+import sys
+from pathlib import Path
+from typing import Any, TypedDict
+
+import torch
+from huggingface_hub import hf_hub_download
+from safetensors.torch import save_file
+from transformers import AutoTokenizer
+
+source_model_id = "deepseek-ai/DeepSeek-V4.1-Flash"
+save_folder = "/tmp/tiny-random/deepseek-v41-flash"
+source_revision = "314113f8a7dadb388215396ba5fbee2c610a04d6"
+implementation_model_id = "deepseek-ai/DeepSeek-V4.1-Flash-1017"
+implementation_revision = "1bae0e1088993b9b82de08f59de5be5a34e42170"
+# DeepSeek-V4.1-Flash is a multimodal (vision + text) MoE model. The released checkpoint uses the
+# inference-stack tensor naming (see the repo's inference/model.py), not HF-transformers naming, so
+# this generator mirrors that layout. The config is nested: text_config + vision_config, with a
+# shared quantization_config (fp8 weights, fp4 routed experts, ue8m0 scales, 32x32 blocks).
+config = {
+    "architectures": [
+        "DeepseekV41ForCausalLM"
+    ],
+    "model_type": "deepseek_v41",
+    "dtype": "bfloat16",
+    "transformers_version": "5.6.0",
+    "bos_token_id": 0,
+    "eos_token_id": 1,
+    "pad_token_id": 2,
+    "image_token_id": 129264,
+    "quantization_config": {
+        "quant_method": "fp8",
+        "activation_scheme": "dynamic",
+        "weight_block_size": [
+            32,
+            32
+        ],
+        "scale_fmt": "ue8m0",
+        "expert_dtype": "fp4"
+    },
+    "text_config": {
+        "model_type": "deepseek_v41_text",
+        "vocab_size": 129280,
+        "hidden_size": 128,
+        # TP=2 leaves 128 columns per rank, the minimum accepted by the
+        # FlashInfer MXFP4 MoE kernel.
+        "moe_intermediate_size": 256,
+        "num_hidden_layers": 6,
+        "num_attention_heads": 4,
+        "num_key_value_heads": 1,
+        # SGLang's DSV4 cache/backend ABI fixes this at 448 non-RoPE + 64 RoPE.
+        "head_dim": 512,
+        "qk_rope_head_dim": 64,
+        "q_lora_rank": 128,
+        "o_lora_rank": 128,
+        "o_groups": 2,
+        "hidden_act": "silu",
+        "swiglu_limit": 10.0,
+        "rms_norm_eps": 1e-20,
+        "attention_bias": False,
+        "attention_dropout": 0.0,
+        "initializer_range": 0.02,
+        "use_cache": True,
+        "tie_word_embeddings": False,
+        "max_position_embeddings": 1048576,
+        "rope_theta": 10000,
+        "rope_scaling": {
+            "rope_type": "yarn",
+            "factor": 16,
+            "beta_fast": 32,
+            "beta_slow": 1,
+            "original_max_position_embeddings": 65536
+        },
+        "n_routed_experts": 32,
+        "n_shared_experts": 1,
+        "num_experts_per_tok": 6,
+        "scoring_func": "sqrtsoftplus",
+        "topk_method": "noaux_tc",
+        "norm_topk_prob": True,
+        "routed_scaling_factor": 1.5,
+        "sliding_window": 128,
+        # one entry per layer (backbone + MTP): 0 = sliding-window only, r = KV compressed r-to-1
+        "compress_ratios": [0, 2, 2, 1, 1, 0, 0, 0, 0],
+        "compress_rope_theta": 160000,
+        # layers that produce (own) the shared compressed KV / index keys
+        "kv_source_layer_ids": [1, 3],
+        "index_source_layer_ids": [1, 3],
+        "index_n_heads": 4,
+        # The SGLang FP4 indexer kernels use a fixed 128-wide key/query layout.
+        "index_head_dim": 128,
+        "index_topk": 512,
+        "candidate_source_layer_id": 3,
+        "candidate_topk_blocks": 2048,
+        "candidate_block_size": 8,
+        "hc_mult": 4,
+        "hc_sinkhorn_iters": 20,
+        "hc_eps": 1e-06,
+        # n-gram engram hash tables added into the residual at a few layers
+        # The final KV source layer must remain Engram-free in SGLang; the
+        # source model likewise carries Engram on an earlier KV source only.
+        "engram_layer_ids": [1, 2],
+        # row counts of the two hash tables; derived from engram_vocab_size via the prime layout
+        # in the repo's engram.py (find_next_prime chain), computed for the tiny vocab below
+        "engram_num_embeddings": [6748, 7592],
+        "engram_max_ngram_size": 4,
+        "engram_vocab_size": 512,
+        "engram_n_heads": 4,
+        "engram_head_dim": 64,
+        "engram_pad_token_id": 2,
+        "engram_compressed_vocab_size": 99092,
+        "num_nextn_predict_layers": 3,
+        "dspark_block_size": 5,
+        "dspark_noise_token_id": 128799,
+        "dspark_target_layer_ids": [3, 4, 5],
+        # Keep the source low-rank Markov head topology while trimming the two
+        # vocabulary-sized BF16 matrices. SGLang only requires a positive rank.
+        "dspark_markov_rank": 16,
+        "dspark_n_routed_experts": 8,
+        "dspark_num_experts_per_tok": 3
+    },
+    "vision_config": {
+        "model_type": "deepseek_v41_vision",
+        "num_hidden_layers": 2,
+        "hidden_size": 64,
+        "num_attention_heads": 4,
+        "intermediate_size": 128,
+        "patch_size": 14,
+        "rope_theta": 10000,
+        "downsample_ratio": 3,
+        "max_image_tokens": 1024,
+        "min_pixels": 295936,
+        "max_wh_ratio": None
+    }
+}
+
+def main():
+    torch.manual_seed(42)
+    Path(save_folder).mkdir(parents=True, exist_ok=True)
+    tokenizer = AutoTokenizer.from_pretrained(
+        source_model_id,
+        revision=source_revision,
+        trust_remote_code=True,
+    )
+    state_dict = generate(config, tokenizer)
+    save_file(state_dict, Path(save_folder) / "model.safetensors")
+    with open(Path(save_folder) / "model.safetensors", "rb") as f:
+        state_dict = f.read()
+        print("Hash: ", hashlib.sha256(state_dict).hexdigest())
+
+    with open(Path(save_folder) / "config.json", "w", encoding="utf-8") as f:
+        json.dump(config, f, indent=2, ensure_ascii=False)
+
+    tokenizer.save_pretrained(save_folder)
+
+    try:
+        from transformers import GenerationConfig
+        generation_config = GenerationConfig.from_pretrained(
+            source_model_id,
+            revision=source_revision,
+            trust_remote_code=True,
+        )
+        generation_config.save_pretrained(save_folder)
+    except Exception as exc:  # noqa: BLE001
+        print(f"Skipping generation config: {exc}")
+
+BF16 = "torch.bfloat16"
+F32 = "torch.float32"
+FP8 = "torch.float8_e4m3fn"
+SCALE = "torch.float8_e8m0fnu"
+I8 = "torch.int8"
+
+class TensorSpec(TypedDict):
+    shape: list[int]
+    dtype: str
+
+Config = dict[str, Any]
+State = dict[str, TensorSpec]
+TensorDict = dict[str, torch.Tensor]
+
+def initialize(state: State, config: Config, init_bound: float) -> TensorDict:
+    tensors: TensorDict = {}
+    scale_dtype = torch.float8_e8m0fnu
+    fp4_boundaries = torch.tensor([0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5.0])
+
+    def scale_for(
+        amax: torch.Tensor, qmax: float, dtype: torch.dtype
+    ) -> torch.Tensor:
+        scale = amax.float().clamp_min(torch.finfo(torch.float32).tiny) / qmax
+        if dtype == scale_dtype:
+            scale = torch.pow(2.0, torch.round(torch.log2(scale)))
+        return scale.to(dtype)
+
+    def fp8_tensor(
+        shape: list[int], scale_shape: list[int], dtype: torch.dtype
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        # block sizes are inferred from the scale shape, so this also handles the engram embedding
+        # table, which is quantized per-row over 32-column blocks (block_out == 1)
+        assert shape[0] % scale_shape[0] == 0 and shape[1] % scale_shape[1] == 0
+        block_out = shape[0] // scale_shape[0]
+        block_in = shape[1] // scale_shape[1]
+        value = torch.empty(shape, dtype=torch.bfloat16).uniform_(
+            -init_bound, init_bound
+        )
+        blocks = value.view(
+            scale_shape[0], block_out, scale_shape[1], block_in
+        ).transpose(1, 2)
+        scales = scale_for(blocks.abs().amax(dim=(-1, -2)), 448, dtype)
+        weight = (
+            (blocks.float() / scales.float()[..., None, None])
+            .clamp(-448, 448)
+            .to(torch.float8_e4m3fn)
+            .transpose(1, 2)
+            .reshape(shape)
+            .contiguous()
+        )
+        return weight, scales
+
+    def fp4_tensors(
+        count: int,
+        shape: list[int],
+        scale_shape: list[int],
+        dtype: torch.dtype,
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        out_dim, packed_in_dim = shape
+        block_out, block_in = config["quantization_config"]["weight_block_size"]
+        assert out_dim % block_out == 0
+        assert packed_in_dim * 2 % block_in == 0
+        assert packed_in_dim == scale_shape[1] * 16
+        value = torch.empty(
+            count, out_dim, packed_in_dim * 2, dtype=torch.bfloat16
+        ).uniform_(-init_bound, init_bound)
+        blocks = value.view(count, out_dim, scale_shape[1], 32)
+        scales = scale_for(blocks.abs().amax(dim=-1), 6, dtype)
+        normalized = (blocks.float() / scales.float()[..., None]).clamp(-6, 6)
+        code = torch.bucketize(normalized.abs(), fp4_boundaries)
+        code += normalized.signbit() * 8
+        code = code.view(count, out_dim, packed_in_dim * 2)
+        packed = (code[..., ::2] | (code[..., 1::2] << 4)).to(torch.uint8)
+        weight = packed.view(torch.int8).contiguous()
+        return weight, scales
+
+    dtype_map: dict[str, torch.dtype] = {
+        BF16: torch.bfloat16,
+        F32: torch.float32,
+        FP8: torch.float8_e4m3fn,
+        SCALE: scale_dtype,
+        I8: torch.int8,
+    }
+    fp4_groups: dict[
+        tuple[tuple[int, ...], tuple[int, ...], torch.dtype], list[str]
+    ] = {}
+    for name, spec in state.items():
+        if spec["dtype"] != I8:
+            continue
+        scale_spec = state[name.replace(".weight", ".scale")]
+        key = (
+            tuple(spec["shape"]),
+            tuple(scale_spec["shape"]),
+            dtype_map[scale_spec["dtype"]],
+        )
+        fp4_groups.setdefault(key, []).append(name)
+
+    fp4_cache: dict[str, tuple[torch.Tensor, torch.Tensor]] = {}
+    max_batch_elements = 4 * 1024 * 1024
+    for (shape_tuple, scale_shape_tuple, dtype), names in fp4_groups.items():
+        shape = list(shape_tuple)
+        scale_shape = list(scale_shape_tuple)
+        logical_elements = shape[0] * shape[1] * 2
+        batch_size = max(1, max_batch_elements // logical_elements)
+        for start in range(0, len(names), batch_size):
+            batch_names = names[start: start + batch_size]
+            weights, scales = fp4_tensors(
+                len(batch_names), shape, scale_shape, dtype
+            )
+            for index, name in enumerate(batch_names):
+                fp4_cache[name] = (
+                    weights[index].clone(),
+                    scales[index].clone(),
+                )
+
+    for name, spec in state.items():
+        if name in tensors:
+            continue
+        shape, dtype = spec["shape"], dtype_map[spec["dtype"]]
+        scale_name = name.replace(".weight", ".scale")
+        if spec["dtype"] == FP8:
+            scale_spec = state[scale_name]
+            scale_type = dtype_map[scale_spec["dtype"]]
+            tensors[name], tensors[scale_name] = fp8_tensor(
+                shape, scale_spec["shape"], scale_type
+            )
+        elif spec["dtype"] == I8:
+            tensors[name], tensors[scale_name] = fp4_cache.pop(name)
+        elif not name.endswith(".scale"):
+            tensors[name] = torch.empty(shape, dtype=dtype).uniform_(
+                -init_bound, init_bound
+            )
+        print(f"{name}: {shape} {dtype}", flush=True)
+    file_size_by_name = {name: tensor.numel() * tensor.element_size() for name, tensor in tensors.items()}
+    total_file_size = sum(file_size_by_name.values())
+    k = 20
+    topk = sorted(file_size_by_name.items(), key=lambda x: x[1], reverse=True)[:k]
+    print(f"File size: {total_file_size / 1024 / 1024:.2f} MB")
+    print(f"Top {k} largest tensors:")
+    for name, size in topk:
+        print(f"  {name}: {size / 1024 / 1024:.2f} MB")
+    return tensors
+
+def generate(config: Config, tokenizer: Any, init_bound: float = 0.2) -> TensorDict:
+    assert init_bound > 0
+    text = config["text_config"]
+    vision = config["vision_config"]
+    quant = config["quantization_config"]
+    rope = text["rope_scaling"]
+
+    # Transformers has no deepseek_v41 registration or auto_map. The source repository instead
+    # ships a complete readable runtime under inference/. Instantiate that implementation on the
+    # meta device so the official modules, rather than a handwritten tensor list, define the tiny
+    # checkpoint structure. Pin the inspected source revision for reproducibility.
+    runtime_files = (
+        "model.py",
+        "engram.py",
+        "vision.py",
+        "kernel.py",
+        "image_processor.py",
+    )
+    runtime_dir = None
+    for filename in runtime_files:
+        file_path = hf_hub_download(
+            implementation_model_id,
+            filename=f"inference/{filename}",
+            revision=implementation_revision,
+        )
+        runtime_dir = str(Path(file_path).parent)
+    assert runtime_dir is not None
+    sys.path.insert(0, runtime_dir)
+    try:
+        from model import ModelArgs, Transformer
+    finally:
+        sys.path.remove(runtime_dir)
+
+    args = ModelArgs(
+        max_batch_size=1,
+        max_seq_len=min(text["max_position_embeddings"], 128),
+        dtype="fp8" if quant["quant_method"] == "fp8" else "bf16",
+        expert_dtype=quant.get("expert_dtype"),
+        vocab_size=text["vocab_size"],
+        dim=text["hidden_size"],
+        moe_inter_dim=text["moe_intermediate_size"],
+        n_layers=text["num_hidden_layers"],
+        n_mtp_layers=text["num_nextn_predict_layers"],
+        n_heads=text["num_attention_heads"],
+        n_routed_experts=text["n_routed_experts"],
+        n_shared_experts=text["n_shared_experts"],
+        n_activated_experts=text["num_experts_per_tok"],
+        score_func=text["scoring_func"],
+        norm_topk_prob=text["norm_topk_prob"],
+        route_scale=text["routed_scaling_factor"],
+        swiglu_limit=text["swiglu_limit"],
+        q_lora_rank=text["q_lora_rank"],
+        head_dim=text["head_dim"],
+        rope_head_dim=text["qk_rope_head_dim"],
+        norm_eps=text["rms_norm_eps"],
+        o_groups=text["o_groups"],
+        o_lora_rank=text["o_lora_rank"],
+        window_size=text["sliding_window"],
+        compress_ratios=tuple(text["compress_ratios"]),
+        kv_source_layers=tuple(text["kv_source_layer_ids"]),
+        index_source_layers=tuple(text["index_source_layer_ids"]),
+        compress_rope_theta=text["compress_rope_theta"],
+        original_seq_len=rope["original_max_position_embeddings"],
+        rope_theta=text["rope_theta"],
+        rope_factor=rope["factor"],
+        beta_fast=rope["beta_fast"],
+        beta_slow=rope["beta_slow"],
+        index_n_heads=text["index_n_heads"],
+        index_head_dim=text["index_head_dim"],
+        index_topk=text["index_topk"],
+        candidate_source_layer=text["candidate_source_layer_id"],
+        candidate_topk_blocks=text["candidate_topk_blocks"],
+        candidate_block_size=text["candidate_block_size"],
+        hc_mult=text["hc_mult"],
+        hc_sinkhorn_iters=text["hc_sinkhorn_iters"],
+        hc_eps=text["hc_eps"],
+        engram_layer_ids=tuple(text["engram_layer_ids"]),
+        engram_num_embeddings=tuple(text["engram_num_embeddings"]),
+        engram_max_ngram_size=text["engram_max_ngram_size"],
+        engram_vocab_size=text["engram_vocab_size"],
+        engram_n_heads=text["engram_n_heads"],
+        engram_head_dim=text["engram_head_dim"],
+        engram_pad_id=text["engram_pad_token_id"],
+        engram_compressed_vocab_size=text["engram_compressed_vocab_size"],
+        vision_n_layers=vision["num_hidden_layers"],
+        vision_dim=vision["hidden_size"],
+        vision_n_heads=vision["num_attention_heads"],
+        vision_inter_dim=vision["intermediate_size"],
+        vision_patch_size=vision["patch_size"],
+        vision_rope_theta=vision["rope_theta"],
+        vision_downsample_ratio=vision["downsample_ratio"],
+        vision_max_n_token=vision["max_image_tokens"],
+        vision_min_pixels=vision["min_pixels"],
+        vision_max_wh_ratio=vision["max_wh_ratio"],
+        image_token_id=config["image_token_id"],
+        dspark_block_size=text["dspark_block_size"],
+        dspark_noise_token_id=text["dspark_noise_token_id"],
+        dspark_target_layer_ids=tuple(text["dspark_target_layer_ids"]),
+        dspark_markov_rank=text["dspark_markov_rank"],
+        dspark_n_routed_experts=text["dspark_n_routed_experts"],
+        dspark_n_activated_experts=text["dspark_num_experts_per_tok"],
+    )
+
+    previous_dtype = torch.get_default_dtype()
+    try:
+        torch.set_default_dtype(torch.bfloat16)
+        with torch.device("meta"):
+            runtime_model = Transformer(args, tokenizer)
+    finally:
+        torch.set_default_dtype(previous_dtype)
+
+    dtype_map = {
+        torch.bfloat16: BF16,
+        torch.float32: F32,
+        torch.float8_e4m3fn: FP8,
+        torch.float8_e8m0fnu: SCALE,
+        torch.float4_e2m1fn_x2: I8,
+    }
+    state: State = {}
+    block_out, block_in = quant["weight_block_size"]
+    for name, tensor in runtime_model.state_dict().items():
+        # Runtime modules tie these to the backbone. The public HF checkpoint stores only the
+        # backbone copies; safetensors cannot duplicate shared storage either.
+        if name.startswith("mtp.") and name.split(".", 2)[-1] in (
+            "embed.weight",
+            "head.weight",
+        ):
+            continue
+
+        shape = list(tensor.shape)
+        dtype = dtype_map[tensor.dtype]
+
+        # inference/convert.py dequantizes wo_a to bf16 for its grouped einsum. Undo that runtime
+        # representation here: the public HF checkpoint stores wo_a as block-quantized fp8.
+        if name.endswith(".attn.wo_a.weight"):
+            assert shape[0] % block_out == 0 and shape[1] % block_in == 0
+            dtype = FP8
+            state[name] = {"shape": shape, "dtype": dtype}
+            state[name.replace(".weight", ".scale")] = {
+                "shape": [shape[0] // block_out, shape[1] // block_in],
+                "dtype": SCALE,
+            }
+            continue
+
+        # The readable runtime promotes these parameters for computation. The published HF
+        # checkpoint keeps them in bf16, as confirmed from all 48 source shard headers.
+        if (
+            name == "head.weight"
+            or name.endswith(".markov_head.head.weight")
+            or name.endswith(".confidence_head.proj.weight")
+            or ".attn.compressor.w" in name
+            or (
+                name.startswith("vision.")
+                and (
+                    name == "vision.norm.weight"
+                    or name.endswith(".norm1.weight")
+                    or name.endswith(".norm2.weight")
+                )
+            )
+        ):
+            dtype = BF16
+
+        state[name] = {"shape": shape, "dtype": dtype}
+
+    del runtime_model
+    return initialize(state, config, init_bound)
+
+main()
+```
+
+</details>
+
+### Test environment:
+
+- Pillow: 12.3.0
+- flashinfer-python: 0.6.15.post1
+- huggingface_hub: 1.27.0
+- numpy: 2.5.2
+- safetensors: 0.8.0
+- sympy: 1.14.0
+- tilelang: 0.1.12
+- torch: 2.13.0+cu126
+- transformers: 5.15.0.dev0
+
+### Change log:
+
+- 2026-09-10: Initial version.
+- 2026-10-20: Replace the handwritten tensor schema with the pinned official V4.1 inference implementation, align all TP=2/DSV4 dimensions, preserve the three-stage DSpark/MTP draft, add the SGLang TP=2 + DSpark launch profile, and document source-versus-tiny structure and validated runtime versions.

chat_template.jinja

@@ -0,0 +1,252 @@
+{#- DeepSeek-V4.1 chat template, ported from encoding/encoding.py in the model repo.
+    Extra kwargs: thinking_mode ("thinking" | "chat", default "thinking"; enable_thinking=true/false is accepted
+    as an alias), drop_thinking (default true), reasoning_effort (int in [1, 100] or "low"/"medium"/"high"/"max",
+    default "high"). Assistant reasoning is read from reasoning_content, or from thinking as an alias. -#}
+{%- set IMG = "<｜deepseek_image｜>" -%}
+{%- set DSML = "｜DSML｜" -%}
+
+{%- if thinking_mode is not defined -%}
+    {%- set thinking_mode = "chat" if (enable_thinking is defined and enable_thinking is not none and not enable_thinking) else "thinking" -%}
+{%- endif -%}
+{%- if thinking_mode not in ["chat", "thinking"] -%}
+    {{- raise_exception("Invalid thinking_mode `" ~ thinking_mode ~ "`, should be \"chat\" or \"thinking\"") -}}
+{%- endif -%}
+{%- if drop_thinking is not defined -%}{%- set drop_thinking = true -%}{%- endif -%}
+{%- if add_generation_prompt is not defined -%}{%- set add_generation_prompt = false -%}{%- endif -%}
+{%- if tools is not defined -%}{%- set tools = none -%}{%- endif -%}
+
+{%- if reasoning_effort is not defined or reasoning_effort is none -%}{%- set reasoning_effort = "high" -%}{%- endif -%}
+{%- set effort_map = {"low": 50, "medium": 62, "high": 75, "max": 100} -%}
+{%- if reasoning_effort is string and reasoning_effort in effort_map -%}
+    {%- set budget = effort_map[reasoning_effort] -%}
+{%- elif reasoning_effort is integer and 1 <= reasoning_effort <= 100 -%}
+    {%- set budget = reasoning_effort -%}
+{%- else -%}
+    {{- raise_exception("Invalid reasoning effort: " ~ reasoning_effort ~ ", should be int within [1,100] or one of \"low\", \"medium\", \"high\", \"max\"") -}}
+{%- endif -%}
+
+{#- Text of text/image blocks joined by blank lines; other block types are dropped. -#}
+{%- macro joined_text(blocks) -%}
+    {%- set jt = namespace(parts=[]) -%}
+    {%- for b in blocks -%}
+        {%- if b.type == "text" -%}
+            {%- if IMG in (b.text or "") -%}{{- raise_exception("Text block contains image placeholder " ~ IMG ~ ". Images should be separate content blocks.") -}}{%- endif -%}
+            {%- set jt.parts = jt.parts + [b.text or ""] -%}
+        {%- elif b.type in ["image", "image_url"] -%}
+            {%- set jt.parts = jt.parts + [IMG] -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- jt.parts | join("\n\n") -}}
+{%- endmacro -%}
+
+{#- Splits/validates "ns::name" against an explicit namespace and returns the qualified name. -#}
+{%- macro qualified_name(name, ns) -%}
+    {%- if ns is mapping -%}{%- set ns = ns.name -%}{%- endif -%}
+    {%- set parts = name.split("::", 1) -%}
+    {%- if parts | length == 2 -%}
+        {%- if ns is not none and ns != parts[0] -%}{{- raise_exception("Conflicting tool namespaces: " ~ ns ~ " != " ~ parts[0]) -}}{%- endif -%}
+        {%- set ns = parts[0] -%}
+        {%- set name = parts[1] -%}
+    {%- endif -%}
+    {%- if "::" in name -%}{{- raise_exception("Tool name must not contain '::': " ~ name) -}}{%- endif -%}
+    {%- if ns is not none and "::" in ns -%}{{- raise_exception("Tool namespace must not contain '::': " ~ ns) -}}{%- endif -%}
+    {{- name if ns is none else ns ~ "::" ~ name -}}
+{%- endmacro -%}
+
+{%- macro render_tools(tool_list) -%}
+    {%- set ts = namespace(schemas=[]) -%}
+    {%- for tool in tool_list -%}
+        {%- set fn = tool.function if tool.function is defined else tool -%}
+        {%- set ns = tool.namespace if tool.namespace is defined and tool.namespace is not none else fn.get("namespace") -%}
+        {%- set schema = namespace(d={}) -%}
+        {%- for k, v in fn.items() if k != "namespace" -%}{%- set schema.d = dict(schema.d, **{k: v}) -%}{%- endfor -%}
+        {%- set schema.d = dict(schema.d, name=qualified_name(fn.name, ns)) -%}
+        {%- if ns is mapping and ns.description -%}
+            {%- set schema.d = dict(schema.d, description=ns.description ~ "\n" ~ (fn.description or "")) -%}
+        {%- endif -%}
+        {%- set ts.schemas = ts.schemas + [schema.d | tojson] -%}
+    {%- endfor -%}
+    {{- "## Tools\n\nYou have access to a set of tools to help answer the user's question. You can invoke tools by writing a \"<｜DSML｜ calls>\" block like the following:\n\n<｜DSML｜ calls>\n<｜DSML｜ invoke name=\"$TOOL_NAME\">\n<｜DSML｜ parameter name=\"$PARAMETER_NAME\" string=\"true|false\">$PARAMETER_VALUE</｜DSML｜ parameter>\n...\n</｜DSML｜ invoke>\n<｜DSML｜ invoke name=\"$TOOL_NAME2\">\n...\n</｜DSML｜ invoke>\n</｜DSML｜ calls>\n\nString parameters should be specified as is and set `string=\"true\"`. For all other types (numbers, booleans, arrays, objects), pass the value in JSON format and set `string=\"false\"`.\n\nIf thinking_mode is enabled (triggered by <think>), you MUST output your complete reasoning inside <think>...</think> BEFORE any tool calls or final response.\n\nOtherwise, output directly after </think> with tool calls or final response.\n\n### Available Tool Schemas\n\n" -}}
+    {{- ts.schemas | join("\n") -}}
+    {{- "\n\nYou MUST strictly follow the above defined tool name and parameter schemas to invoke tool calls.\n" -}}
+{%- endmacro -%}
+
+{#- 1. Merge tool results (and consecutive user turns) into user messages made of content blocks. -#}
+{%- set mns = namespace(list=[]) -%}
+{%- for msg in messages -%}
+    {%- set content = msg.get("content") -%}
+    {%- set is_blocks = content is iterable and content is not string and content is not mapping -%}
+    {%- if content is string and IMG in content -%}
+        {{- raise_exception("Message content contains image special token " ~ IMG ~ ". Images should be provided as image content blocks.") -}}
+    {%- endif -%}
+    {%- set rc = msg.reasoning_content if msg.get("reasoning_content") is not none else msg.get("thinking") -%}
+    {%- if rc is string and IMG in rc -%}
+        {{- raise_exception("reasoning_content contains image special token " ~ IMG) -}}
+    {%- endif -%}
+    {%- set prev = mns.list[-1] if mns.list else none -%}
+    {%- if msg.role == "tool" -%}
+        {%- set tblock = {"type": "tool_result", "tool_use_id": msg.get("tool_call_id", ""), "content": joined_text(content) if is_blocks else msg.get("content", "")} -%}
+        {%- if prev is not none and prev.role == "user" -%}
+            {%- set mns.list = mns.list[:-1] + [dict(prev, content_blocks=prev.content_blocks + [tblock])] -%}
+        {%- else -%}
+            {%- set mns.list = mns.list + [{"role": "user", "content_blocks": [tblock]}] -%}
+        {%- endif -%}
+    {%- elif msg.role == "user" -%}
+        {%- set blocks = content if is_blocks else [{"type": "text", "text": content or ""}] -%}
+        {%- if prev is not none and prev.role == "user" and prev.get("task") is none -%}
+            {%- set mns.list = mns.list[:-1] + [dict(prev, content_blocks=prev.content_blocks + blocks)] -%}
+        {%- else -%}
+            {%- set mns.list = mns.list + [dict(msg, content_blocks=blocks)] -%}
+        {%- endif -%}
+    {%- else -%}
+        {%- set mns.list = mns.list + [msg] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- Top-level tools attach to a leading system message, or to a new empty one. -#}
+{%- if tools -%}
+    {%- if mns.list and mns.list[0].role == "system" -%}
+        {%- if not mns.list[0].get("tools") -%}{%- set mns.list = [dict(mns.list[0], tools=tools)] + mns.list[1:] -%}{%- endif -%}
+    {%- else -%}
+        {%- set mns.list = [{"role": "system", "content": "", "tools": tools}] + mns.list -%}
+    {%- endif -%}
+{%- endif -%}
+
+{#- 2. Reorder tool results to match the tool_calls order of the preceding assistant message. -#}
+{%- set sns = namespace(list=[], ids=[]) -%}
+{%- for m in mns.list -%}
+    {%- if m.role == "assistant" and m.get("tool_calls") -%}
+        {%- set sns.ids = [] -%}
+        {%- for tc in m.tool_calls -%}
+            {%- set sns.ids = sns.ids + [tc.get("id") or tc.get("function", {}).get("id", "")] -%}
+        {%- endfor -%}
+        {%- set sns.list = sns.list + [m] -%}
+    {%- elif m.role == "user" and m.content_blocks -%}
+        {%- set tb = namespace(blocks=[]) -%}
+        {%- for b in m.content_blocks if b.type == "tool_result" -%}
+            {%- set order = namespace(idx=0) -%}
+            {%- for tc_id in sns.ids -%}
+                {%- if tc_id and tc_id == b.get("tool_use_id", "") -%}{%- set order.idx = loop.index0 -%}{%- endif -%}
+            {%- endfor -%}
+            {%- set tb.blocks = tb.blocks + [dict(b, _order=order.idx)] -%}
+        {%- endfor -%}
+        {%- if tb.blocks | length > 1 -%}
+            {%- set sorted_blocks = tb.blocks | sort(attribute="_order") -%}
+            {%- set rb = namespace(blocks=[], i=0) -%}
+            {%- for b in m.content_blocks -%}
+                {%- if b.type == "tool_result" -%}
+                    {%- set rb.blocks = rb.blocks + [sorted_blocks[rb.i]] -%}
+                    {%- set rb.i = rb.i + 1 -%}
+                {%- else -%}
+                    {%- set rb.blocks = rb.blocks + [b] -%}
+                {%- endif -%}
+            {%- endfor -%}
+            {%- set sns.list = sns.list + [dict(m, content_blocks=rb.blocks)] -%}
+        {%- else -%}
+            {%- set sns.list = sns.list + [m] -%}
+        {%- endif -%}
+    {%- else -%}
+        {%- set sns.list = sns.list + [m] -%}
+    {%- endif -%}
+{%- endfor -%}
+{%- set msgs = sns.list -%}
+
+{#- Mid-conversation system messages count as user turns. Tools anywhere disable drop_thinking. -#}
+{%- set state = namespace(last_user=-1, has_tools=false) -%}
+{%- for m in msgs -%}
+    {%- if m.role == "user" or (m.role == "system" and loop.index0 > 0) -%}{%- set state.last_user = loop.index0 -%}{%- endif -%}
+    {%- if m.get("tools") -%}{%- set state.has_tools = true -%}{%- endif -%}
+{%- endfor -%}
+{%- set effective_drop = drop_thinking and not state.has_tools -%}
+{%- set thinking = thinking_mode == "thinking" -%}
+
+{{- "<｜begin▁of▁sentence｜>" -}}
+{#- 3. Render each message. -#}
+{%- for m in msgs -%}
+    {%- set idx = loop.index0 -%}
+    {%- set content = m.get("content") -%}
+    {%- if content is iterable and content is not string and content is not mapping -%}{%- set content = joined_text(content) -%}{%- endif -%}
+
+    {%- if idx == 0 -%}
+        {%- if thinking or m.role == "system" -%}{{- "<｜System｜>" -}}{%- endif -%}
+        {%- if thinking -%}{{- "Reasoning Effort: " ~ budget ~ " (range 1-100, the higher the value, the more thorough the reasoning)\n\n" -}}{%- endif -%}
+    {%- endif -%}
+
+    {%- if m.role == "system" -%}
+        {%- if idx > 0 -%}{{- "<｜System｜>" -}}{%- endif -%}
+        {{- content or "" -}}
+        {%- if m.get("tools") -%}{{- "\n\n" ~ render_tools(m.tools) -}}{%- endif -%}
+        {%- if m.get("response_format") -%}
+            {{- "\n\n## Response Format:\n\nYou MUST strictly adhere to the following schema to reply:\n" ~ (m.response_format | tojson) -}}
+        {%- endif -%}
+
+    {%- elif m.role == "user" -%}
+        {{- "<｜User｜>" -}}
+        {%- set up = namespace(parts=[]) -%}
+        {%- for b in m.content_blocks -%}
+            {%- if b.type == "text" -%}
+                {%- if IMG in (b.text or "") -%}{{- raise_exception("Text block contains image placeholder " ~ IMG ~ ". Images should be separate content blocks.") -}}{%- endif -%}
+                {%- set up.parts = up.parts + [b.text or ""] -%}
+            {%- elif b.type in ["image", "image_url"] -%}
+                {%- set up.parts = up.parts + [IMG] -%}
+            {%- elif b.type == "tool_result" -%}
+                {%- set tc = b.get("content", "") -%}
+                {%- if tc is iterable and tc is not string and tc is not mapping -%}
+                    {%- set sub = namespace(parts=[]) -%}
+                    {%- for s in tc -%}
+                        {%- if s.type == "text" -%}{%- set sub.parts = sub.parts + [s.get("text", "")] -%}
+                        {%- elif s.type in ["image", "image_url"] -%}{%- set sub.parts = sub.parts + [IMG] -%}
+                        {%- else -%}{%- set sub.parts = sub.parts + ["[Unsupported " ~ s.type ~ "]"] -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                    {%- set tc = sub.parts | join("\n\n") -%}
+                {%- endif -%}
+                {%- set up.parts = up.parts + ["<tool_result>" ~ tc ~ "</tool_result>"] -%}
+            {%- else -%}
+                {%- set up.parts = up.parts + ["[Unsupported " ~ b.type ~ "]"] -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {{- up.parts | join("\n\n") -}}
+
+    {%- elif m.role == "latest_reminder" -%}
+        {{- "<｜latest_reminder｜>" ~ content -}}
+
+    {%- elif m.role == "assistant" -%}
+        {%- set prev_has_task = idx > 0 and msgs[idx - 1].get("task") is not none -%}
+        {%- if thinking and not prev_has_task and (not effective_drop or idx > state.last_user) -%}
+            {{- ((m.reasoning_content if m.get("reasoning_content") is not none else m.get("thinking")) or "") ~ "</think>" -}}
+        {%- endif -%}
+        {{- content or "" -}}
+        {%- if m.get("tool_calls") -%}
+            {%- set tcs = namespace(lines=[]) -%}
+            {%- for tc in m.tool_calls -%}
+                {%- set fn = tc.function -%}
+                {%- set params = namespace(lines=[]) -%}
+                {%- set args = fn.arguments if fn.arguments is mapping else {"arguments": fn.arguments} -%}
+                {%- for k, v in args.items() -%}
+                    {%- set params.lines = params.lines + ["<" ~ DSML ~ " parameter name=\"" ~ k ~ "\" string=\"" ~ ("true" if v is string else "false") ~ "\">" ~ (v if v is string else v | tojson) ~ "</" ~ DSML ~ " parameter>"] -%}
+                {%- endfor -%}
+                {%- set name = qualified_name(fn.name, tc.get("namespace") or fn.get("namespace")) -%}
+                {%- set tcs.lines = tcs.lines + ["<" ~ DSML ~ " invoke name=\"" ~ name ~ "\">\n" ~ params.lines | join("\n") ~ "\n</" ~ DSML ~ " invoke>"] -%}
+            {%- endfor -%}
+            {{- "\n\n<" ~ DSML ~ " calls>\n" ~ tcs.lines | join("\n") ~ "\n</" ~ DSML ~ " calls>" -}}
+        {%- endif -%}
+        {%- if not m.get("wo_eos", false) -%}{{- "<｜end▁of▁sentence｜>" -}}{%- endif -%}
+
+    {%- else -%}
+        {{- raise_exception("Unknown role: " ~ m.role) -}}
+    {%- endif -%}
+
+    {#- Transition tokens, only when the next message is an assistant/latest_reminder turn or this is the last one. -#}
+    {%- if loop.last or msgs[idx + 1].role in ["assistant", "latest_reminder"] -%}
+        {%- set task = m.get("task") -%}
+        {%- if task is not none -%}
+            {%- set task_tokens = {"action": "<｜action｜>", "query": "<｜query｜>", "authority": "<｜authority｜>", "domain": "<｜domain｜>", "title": "<｜title｜>", "read_url": "<｜read_url｜>"} -%}
+            {%- if task not in task_tokens -%}{{- raise_exception("Invalid task: '" ~ task ~ "'") -}}{%- endif -%}
+            {%- if task == "action" -%}{{- "<｜Assistant｜>" ~ ("<think>" if thinking else "</think>") -}}{%- endif -%}
+            {{- task_tokens[task] -}}
+        {%- elif (m.role == "user" or (m.role == "system" and idx > 0)) and (add_generation_prompt or not loop.last) -%}
+            {{- "<｜Assistant｜>" ~ ("<think>" if thinking and (not effective_drop or idx >= state.last_user) else "</think>") -}}
+        {%- endif -%}
+    {%- endif -%}
+{%- endfor -%}

config.json

@@ -0,0 +1,128 @@
+{
+  "architectures": [
+    "DeepseekV41ForCausalLM"
+  ],
+  "model_type": "deepseek_v41",
+  "dtype": "bfloat16",
+  "transformers_version": "5.6.0",
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 2,
+  "image_token_id": 129264,
+  "quantization_config": {
+    "quant_method": "fp8",
+    "activation_scheme": "dynamic",
+    "weight_block_size": [
+      32,
+      32
+    ],
+    "scale_fmt": "ue8m0",
+    "expert_dtype": "fp4"
+  },
+  "text_config": {
+    "model_type": "deepseek_v41_text",
+    "vocab_size": 129280,
+    "hidden_size": 128,
+    "moe_intermediate_size": 256,
+    "num_hidden_layers": 6,
+    "num_attention_heads": 4,
+    "num_key_value_heads": 1,
+    "head_dim": 512,
+    "qk_rope_head_dim": 64,
+    "q_lora_rank": 128,
+    "o_lora_rank": 128,
+    "o_groups": 2,
+    "hidden_act": "silu",
+    "swiglu_limit": 10.0,
+    "rms_norm_eps": 1e-20,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "initializer_range": 0.02,
+    "use_cache": true,
+    "tie_word_embeddings": false,
+    "max_position_embeddings": 1048576,
+    "rope_theta": 10000,
+    "rope_scaling": {
+      "rope_type": "yarn",
+      "factor": 16,
+      "beta_fast": 32,
+      "beta_slow": 1,
+      "original_max_position_embeddings": 65536
+    },
+    "n_routed_experts": 32,
+    "n_shared_experts": 1,
+    "num_experts_per_tok": 6,
+    "scoring_func": "sqrtsoftplus",
+    "topk_method": "noaux_tc",
+    "norm_topk_prob": true,
+    "routed_scaling_factor": 1.5,
+    "sliding_window": 128,
+    "compress_ratios": [
+      0,
+      2,
+      2,
+      1,
+      1,
+      0,
+      0,
+      0,
+      0
+    ],
+    "compress_rope_theta": 160000,
+    "kv_source_layer_ids": [
+      1,
+      3
+    ],
+    "index_source_layer_ids": [
+      1,
+      3
+    ],
+    "index_n_heads": 4,
+    "index_head_dim": 128,
+    "index_topk": 512,
+    "candidate_source_layer_id": 3,
+    "candidate_topk_blocks": 2048,
+    "candidate_block_size": 8,
+    "hc_mult": 4,
+    "hc_sinkhorn_iters": 20,
+    "hc_eps": 1e-06,
+    "engram_layer_ids": [
+      1,
+      2
+    ],
+    "engram_num_embeddings": [
+      6748,
+      7592
+    ],
+    "engram_max_ngram_size": 4,
+    "engram_vocab_size": 512,
+    "engram_n_heads": 4,
+    "engram_head_dim": 64,
+    "engram_pad_token_id": 2,
+    "engram_compressed_vocab_size": 99092,
+    "num_nextn_predict_layers": 3,
+    "dspark_block_size": 5,
+    "dspark_noise_token_id": 128799,
+    "dspark_target_layer_ids": [
+      3,
+      4,
+      5
+    ],
+    "dspark_markov_rank": 16,
+    "dspark_n_routed_experts": 8,
+    "dspark_num_experts_per_tok": 3
+  },
+  "vision_config": {
+    "model_type": "deepseek_v41_vision",
+    "num_hidden_layers": 2,
+    "hidden_size": 64,
+    "num_attention_heads": 4,
+    "intermediate_size": 128,
+    "patch_size": 14,
+    "rope_theta": 10000,
+    "downsample_ratio": 3,
+    "max_image_tokens": 1024,
+    "min_pixels": 295936,
+    "max_wh_ratio": null
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53aae999fdcacd310c400a2fc3112bb68ce35588441784c75dca5820abf0f35b
+size 96712592

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": false,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}