| --- |
| license: mit |
| base_model: |
| - Qwen/Qwen3.8-Flash-Next |
| tags: |
| - qwen4_exp |
| - fixture |
| - testing |
| - moe |
| - linear-attention |
| pipeline_tag: text-generation |
| --- |
| |
| # Qwen3.8-Flash-Next tiny architecture fixture (qwen4_exp) |
| |
| A ~0.29M-parameter random-init TEXT checkpoint plus a reduced config that keeps the |
| real **Qwen3.8-Flash-Next** `qwen4_exp` field names, so loaders, quant planners, |
| and CI jobs can exercise the new architecture's config parsing, expert-table |
| sizing, and safetensors load path without the real weights. |
|
|
| ## What this is |
| - Base model: `Qwen/Qwen3.8-Flash-Next` (released 2026-08-24), |
| `model_type: qwen4_exp`, `architectures: ["Qwen4ExpForConditionalGeneration"]`. |
| The base is a large multimodal MoE with a hybrid linear+full attention stack, |
| ngram vocab, an attention indexer, PLE, hyper-connections, and an MTP head, so it |
| cannot be instantiated in a unit test, in CI, or on a laptop. |
| - What this fixture is: a byte-reproducible **random-init** text checkpoint plus a |
| reduced config that preserves the real top-level wrapper fields (`model_type`, |
| `architectures`, `image_token_id`, `video_token_id`, `vision_start/end_token_id`, |
| `language_model_only`, `text_config`, `vision_config`) and the real |
| `qwen4_exp_text` field names inside `text_config` — including `layer_types` |
| (the linear/full attention schedule), `linear_*`, `ngram_*`, `indexer_*`, |
| `ple_*`, `hc_*`, the nested `mtp` block, and `rope_parameters`. |
| - How it is changed from the base: same schema, tiny geometry, float32, and a |
| **reduced standard-attention + MoE tensor set**. It is a schema/load fixture, not |
| a quantization or a distilled model. |
| - What it is **not**: not trained, not distilled, not a quality or benchmark claim. |
| - Why it is useful: it lets you test config parsing, weight-name mapping, |
| expert-table sizing, router/top-k bookkeeping, and safetensors load paths in |
| milliseconds, and it exercises the `qwen4_exp` multimodal wrapper shape that a |
| plain text-only fixture does not. |
|
|
| ## Fixture geometry (what this checkpoint actually contains) |
| | Field | Value | |
| |---|---| |
| | num_hidden_layers | 4 | |
| | hidden_size | 64 | |
| | num_attention_heads / num_key_value_heads | 4 / 2 | |
| | head_dim | 16 | |
| | layer_types | linear, linear, linear, full | |
| | num_experts | 8 | |
| | num_experts_per_tok | 2 | |
| | shared_expert_intermediate_size | 32 | |
| | moe_intermediate_size | 32 | |
| | dtype | float32 | |
| | vocab_size | 256 | |
|
|
| Total: 138 tensors, 1,157,376 data bytes = 289,344 float32 parameters. |
|
|
| ## Intentional omissions (documented, not silent) |
| - **Text-only.** No vision encoder or projector tensors; `vision_config` is a |
| placeholder. A full multimodal loader must supply vision/projector tensors. |
| - The tensor set is a **reduced standard-attention + MoE convention**. The real |
| model's linear-attention (conv/ssm) tensors, ngram-embedding parts, attention |
| indexer, PLE, hyper-connection, and MTP tensors are NOT included. |
| - No `lm_head` tensor; a loader must tie to `model.embed_tokens.weight` or supply |
| its own head. |
| - Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer. |
|
|
| ## Verification actually performed (stdlib only, no torch in this environment) |
| - safetensors header parses: 138 tensors, contiguous `data_offsets`, header padded |
| to 8-byte alignment; `__metadata__` records the generator string. |
| - `config.json` parses; `model_type` is `qwen4_exp` with a nested `qwen4_exp_text` |
| `text_config` carrying the real field names listed above. |
| - `checksums.txt` records the SHA-256 of every tensor blob. |
| - Deterministic regeneration: SplitMix64 seed 20260903, Box-Muller normals, scale |
| 0.02, float32 row-major, consumed in sorted-name order. |
| - Generator script is included as `build_fixture.py` so the folder can be rebuilt |
| and diffed. |
|
|
| **Not yet verified:** loading under a specific `transformers` version (no |
| torch/transformers in the build environment), and whether |
| `Qwen4ExpForConditionalGeneration` accepts this reduced text-only geometry without |
| the linear-attention / ngram / indexer / PLE / MTP tensors. Treat those as open |
| until run against a real install. |
|
|
| ## How to run it |
| Read the tensors with the standard library (no torch needed, matching how this was |
| built): |
|
|
| ```python |
| import json, struct |
| with open("model.safetensors", "rb") as f: |
| n = struct.unpack("<Q", f.read(8))[0] |
| header = json.loads(f.read(n)) |
| # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n |
| ``` |
|
|
| Or with the `safetensors` package: |
|
|
| ```python |
| from safetensors.torch import load_file |
| tensors = load_file("model.safetensors") # {name: tensor} |
| ``` |
|
|
| To exercise a real loader, build a config from `config.json` (the `qwen4_exp` |
| model type; use `AutoConfig.from_pretrained(..., trust_remote_code=True)` where |
| needed) and feed these weights in. There is no `lm_head` tensor, no vision stack, |
| and the tokenizer files are placeholders, so supply your own head/tokenizer/vision. |
|
|
| ## License |
| The generated fixture content (random weights, config, scripts) is released under |
| MIT (see `LICENSE`). The `qwen4_exp` architecture and config schema belong to the |
| base model `Qwen/Qwen3.8-Flash-Next` under its own terms, which were **not** |
| independently re-verified this cycle — check the base repository before |
| redistribution. |
|
|
| ## Citation |
| Qwen Team, Qwen3.8-Flash-Next, 2026. |
|
|
| --- |
|
|
| Buy me a beer: |
| **BTC** `bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p` |
|
|
