| # Minimal inference |
|
|
| A readable reference implementation rather than a production serving engine. The |
| model code covers the vision encoder and aligner, sliding-window plus compressed |
| sparse attention with its two-level indexer, engram n-gram lookups, MoE, |
| Hyper-Connections, and the DSpark forward path. Generation itself is plain |
| autoregressive sampling. |
|
|
| ## Install |
|
|
| ```bash |
| python -m pip install -r requirements.txt |
| ``` |
|
|
| ## Convert Hugging Face weights |
|
|
| The runtime uses one converted checkpoint file per tensor-parallel rank. From |
| this directory: |
|
|
| ```bash |
| export HF_CKPT_PATH=/path/to/DeepSeek-V4.1-Flash-HF |
| export SAVE_PATH=/path/to/DeepSeek-V4.1-Flash-TP8 |
| export MP=8 |
| |
| python convert.py \ |
| --hf-ckpt-path "${HF_CKPT_PATH}" \ |
| --save-path "${SAVE_PATH}" \ |
| --model-parallel "${MP}" \ |
| --expert-dtype fp4 \ |
| --tokenizer-path "${HF_CKPT_PATH}" |
| ``` |
|
|
| Expert counts are inferred from the weight names, so they do not need to be |
| passed. `--tokenizer-path` points at whichever directory holds `tokenizer.json` |
| and `tokenizer_config.json`; they are copied into the converted checkpoint. |
|
|
| ## Run the equivalent TXT and JSON examples |
|
|
| ```bash |
| export CKPT_PATH=/path/to/DeepSeek-V4.1-Flash-TP8 |
| export MP=8 |
| |
| INPUT_FILE=examples/example.txt ./run.sh |
| INPUT_FILE=examples/example_harmony.json ./run.sh |
| ``` |
|
|
| The two files express the same interleaved two-image prompt, so they produce |
| identical encoded prompts and input token IDs. |
|
|
| For interactive chat: |
|
|
| ```bash |
| torchrun --nproc-per-node "${MP}" generate.py \ |
| --ckpt-path "${CKPT_PATH}" \ |
| --config config.json \ |
| --interactive \ |
| --temperature 0.6 |
| ``` |
|
|
| For multi-node execution, pass the usual `torchrun --nnodes`, `--node-rank`, |
| `--master-addr`, and `--master-port` arguments before `generate.py`. |
|
|
| ## Self-test |
|
|
| `model.py` builds a small model from the `ModelArgs` defaults and runs a prefill |
| plus 22 decode steps, exercising the real dense-fp8 / MoE-fp4 kernels. Weights |
| are uninitialized, so it checks shapes and kernel plumbing, not numerics: |
|
|
| ```bash |
| python model.py |
| ``` |
|
|
