Add the Multi-Token Prediction head and a model card

README.md

@@ -1,3 +1,68 @@
 ---
 license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+  - tiny-random
+  - testing
 ---
+
+# tiny-random-qwen38-flash-next
+
+A tiny model with random weights and the architecture of
+[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) (`qwen4_exp`, `Qwen4ExpForConditionalGeneration`),
+for testing (optimum-intel / OpenVINO). Its outputs are not meaningful. It needs transformers >= 5.17.
+
+## Architecture
+
+Every component type of the real model is present, with the same structure and smaller sizes:
+
+| Component | Tiny model | Qwen3.8-Flash-Next |
+|---|---|---|
+| Layers | 8 = 2 × (3 Gated DeltaNet + 1 Qwen Sparse Attention) | 48 = 12 × (3 + 1) |
+| Hidden size | 16 | 2560 |
+| Qwen Sparse Attention (QSA) | 4 query / 2 KV heads × 64; indexer 4 heads × 16, blocks of 4 tokens, budget 8 tokens | 24 / 2 × 256; indexer 4 × 128, blocks of 4, budget 2048 |
+| Gated DeltaNet | 2 key / 6 value heads × 8 | 16 / 48 × 128 |
+| MoE | 16 experts, top-4, shared expert | 512 experts, top-10, shared expert |
+| Hyper-connections | 4 streams, low rank 8 | 4 streams, low rank 320 |
+| N-gram PLE (layer 2) | bi- and tri-grams, 2 heads per order, prime vocabularies from 1000, 4 shards | 8 heads per order, prime vocabularies from 2·10^7, 128 shards |
+| Multi-Token Prediction head | 1 QSA layer (`mtp.*`) | 1 QSA layer (`mtp.*`) |
+| Vision encoder | depth 2, hidden 32 | depth 27, hidden 1152 |
+| Vocabulary, tokenizer, chat template, processors | the real ones | |
+
+The selection budget is small so that QSA is really sparse on short prompts.
+
+The checkpoint keeps the layout of the real one:
+- fused 3D `experts.gate_up_proj` / `experts.down_proj`;
+- the n-gram table split into `ngram_embedding.shard_*`;
+- the MTP head under `mtp.`.
+
+transformers does not build the MTP head (`_keys_to_ignore_on_load_unexpected = ["^mtp.*"]`) and ignores its weights
+when loading. Its structure follows vLLM (`vllm/models/qwen4_exp`):
+- `fc_embedding` projects the next token's embedding;
+- a shared `fc_hidden` projects the main model's pre-final-mixer streams;
+- the projected embedding is added to every stream;
+- one QSA decoder layer and the head's `hyper_connection_mixer` follow, then the shared `lm_head`.
+
+## Initialization
+
+The default initialization (std 0.02) gives nearly uniform logits. Greedy tokens then depend on rounding noise, and
+comparing runtimes means little. The weights are therefore drawn with more variance:
+- layers: std 0.25;
+- embeddings: std 1;
+- `lm_head`: std 2;
+- MoE routers: std 1.
+
+The same scheme is used for the MTP head.
+
+The model was checked on text, image and video prompts. For both the main model and the MTP head:
+- outputs are finite, with logits std about 4 and clear greedy top-2 margins (median above 0.3);
+- 32 greedy tokens are all or nearly all distinct, with no repetition runs;
+- all experts are used;
+- QSA drops 23–93% of the visible tokens.
+
+For the main model, the n-gram PLE contributes 18–59% of the residual-stream norm.
+
+For the MTP head:
+- its predictions depend on both of its inputs;
+- a chained second draft step stays healthy.

config.json

@@ -45,7 +45,16 @@
     "max_position_embeddings": 262144,
     "model_type": "qwen4_exp_text",
     "moe_intermediate_size": 16,
-    "mtp_num_hidden_layers": 0,
+    "mtp": {
+      "hybrid": true,
+      "layer_types": [
+        "full_attention"
+      ],
+      "mtp_use_hidden_state_from_layer": null,
+      "num_hidden_layers": 1,
+      "rope_theta": 10000000
+    },
+    "mtp_num_hidden_layers": 1,
     "mtp_use_dedicated_embeddings": false,
     "ngram_size": 3,
     "ngram_vocab_size_base": 1000,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0108c317c660ae4fbbe3caec3fd90e92ee8821881a0868a05a7028fdca8674b9
-size 33262920
+oid sha256:56d05794d740324ed97e1a8d5183d9f8b9bab7f80c61c49396eb42df6b6428b5
+size 33408504