File size: 5,407 Bytes
018d2c0 24f46e1 018d2c0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 | ---
license: mit
base_model:
- Qwen/Qwen3.8-Flash-Next
tags:
- qwen4_exp
- fixture
- testing
- moe
- linear-attention
pipeline_tag: text-generation
---
# Qwen3.8-Flash-Next tiny architecture fixture (qwen4_exp)
A ~0.29M-parameter random-init TEXT checkpoint plus a reduced config that keeps the
real **Qwen3.8-Flash-Next** `qwen4_exp` field names, so loaders, quant planners,
and CI jobs can exercise the new architecture's config parsing, expert-table
sizing, and safetensors load path without the real weights.
## What this is
- Base model: `Qwen/Qwen3.8-Flash-Next` (released 2026-08-24),
`model_type: qwen4_exp`, `architectures: ["Qwen4ExpForConditionalGeneration"]`.
The base is a large multimodal MoE with a hybrid linear+full attention stack,
ngram vocab, an attention indexer, PLE, hyper-connections, and an MTP head, so it
cannot be instantiated in a unit test, in CI, or on a laptop.
- What this fixture is: a byte-reproducible **random-init** text checkpoint plus a
reduced config that preserves the real top-level wrapper fields (`model_type`,
`architectures`, `image_token_id`, `video_token_id`, `vision_start/end_token_id`,
`language_model_only`, `text_config`, `vision_config`) and the real
`qwen4_exp_text` field names inside `text_config` — including `layer_types`
(the linear/full attention schedule), `linear_*`, `ngram_*`, `indexer_*`,
`ple_*`, `hc_*`, the nested `mtp` block, and `rope_parameters`.
- How it is changed from the base: same schema, tiny geometry, float32, and a
**reduced standard-attention + MoE tensor set**. It is a schema/load fixture, not
a quantization or a distilled model.
- What it is **not**: not trained, not distilled, not a quality or benchmark claim.
- Why it is useful: it lets you test config parsing, weight-name mapping,
expert-table sizing, router/top-k bookkeeping, and safetensors load paths in
milliseconds, and it exercises the `qwen4_exp` multimodal wrapper shape that a
plain text-only fixture does not.
## Fixture geometry (what this checkpoint actually contains)
| Field | Value |
|---|---|
| num_hidden_layers | 4 |
| hidden_size | 64 |
| num_attention_heads / num_key_value_heads | 4 / 2 |
| head_dim | 16 |
| layer_types | linear, linear, linear, full |
| num_experts | 8 |
| num_experts_per_tok | 2 |
| shared_expert_intermediate_size | 32 |
| moe_intermediate_size | 32 |
| dtype | float32 |
| vocab_size | 256 |
Total: 138 tensors, 1,157,376 data bytes = 289,344 float32 parameters.
## Intentional omissions (documented, not silent)
- **Text-only.** No vision encoder or projector tensors; `vision_config` is a
placeholder. A full multimodal loader must supply vision/projector tensors.
- The tensor set is a **reduced standard-attention + MoE convention**. The real
model's linear-attention (conv/ssm) tensors, ngram-embedding parts, attention
indexer, PLE, hyper-connection, and MTP tensors are NOT included.
- No `lm_head` tensor; a loader must tie to `model.embed_tokens.weight` or supply
its own head.
- Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.
## Verification actually performed (stdlib only, no torch in this environment)
- safetensors header parses: 138 tensors, contiguous `data_offsets`, header padded
to 8-byte alignment; `__metadata__` records the generator string.
- `config.json` parses; `model_type` is `qwen4_exp` with a nested `qwen4_exp_text`
`text_config` carrying the real field names listed above.
- `checksums.txt` records the SHA-256 of every tensor blob.
- Deterministic regeneration: SplitMix64 seed 20260903, Box-Muller normals, scale
0.02, float32 row-major, consumed in sorted-name order.
- Generator script is included as `build_fixture.py` so the folder can be rebuilt
and diffed.
**Not yet verified:** loading under a specific `transformers` version (no
torch/transformers in the build environment), and whether
`Qwen4ExpForConditionalGeneration` accepts this reduced text-only geometry without
the linear-attention / ngram / indexer / PLE / MTP tensors. Treat those as open
until run against a real install.
## How to run it
Read the tensors with the standard library (no torch needed, matching how this was
built):
```python
import json, struct
with open("model.safetensors", "rb") as f:
n = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(n))
# header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n
```
Or with the `safetensors` package:
```python
from safetensors.torch import load_file
tensors = load_file("model.safetensors") # {name: tensor}
```
To exercise a real loader, build a config from `config.json` (the `qwen4_exp`
model type; use `AutoConfig.from_pretrained(..., trust_remote_code=True)` where
needed) and feed these weights in. There is no `lm_head` tensor, no vision stack,
and the tokenizer files are placeholders, so supply your own head/tokenizer/vision.
## License
The generated fixture content (random weights, config, scripts) is released under
MIT (see `LICENSE`). The `qwen4_exp` architecture and config schema belong to the
base model `Qwen/Qwen3.8-Flash-Next` under its own terms, which were **not**
independently re-verified this cycle — check the base repository before
redistribution.
## Citation
Qwen Team, Qwen3.8-Flash-Next, 2026.
---
Buy me a beer:
**BTC** `bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p`
|