Create README.md

README.md

@@ -0,0 +1,322 @@
+---
+license: other
+license_name: deepseek
+license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark/blob/main/LICENSE
+base_model: deepseek-ai/DeepSeek-V4-Flash-DSpark
+language:
+  - en
+  - zh
+tags:
+  - deepseek
+  - deepseek-v4
+  - moe
+  - mxfp4
+  - mla
+  - speculative-decoding
+  - dspark
+  - amd-npu
+  - xdna2
+  - ryzen-ai
+  - fst
+pipeline_tag: text-generation
+library_name: fst
+---
+
+# Model Card for DeepSeek-V4-Flash-DSpark-FST
+
+This is the **FaStar `.fst` container** of DeepSeek V4 Flash (DSpark), a 284-billion-parameter
+Mixture-of-Experts model. It is a **format/quantization conversion** of
+[`deepseek-ai/DeepSeek-V4-Flash-DSpark`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark),
+not a retrained or fine-tuned model. The `.fst` format packs the model for execution on the
+**AMD Ryzen AI 9 365 XDNA2 NPU** via the [FaStar](https://github.com/RaffaelloMolinari/FaStar)
+inference engine, which spills the ~150 GB of expert weights across **SSD → RAM → NPU scratch
+buffers** using *expert virtual memory*.
+
+## Model Details
+
+### Model Description
+
+- **Developed by:** Original model by [DeepSeek-AI](https://github.com/deepseek-ai); `.fst`
+  conversion and FaStar inference engine by Raffaello Molinari.
+- **Funded by [optional]:** Independent research/engineering project.
+- **Shared by [optional]:** Raffaello Molinari.
+- **Model type:** Decoder-only Transformer with Mixture-of-Experts (MoE) and Multi-head Latent
+  Attention (MLA), MXFP4-quantized, with a DSpark draft model for speculative decoding.
+- **Language(s) (NLP):** English, Chinese (DeepSeek V4 is multilingual; primarily en/zh).
+- **License:** DeepSeek Model License (inherited from the upstream checkpoint — see `license_link`).
+- **Finetuned from model [optional]:** Not fine-tuned. Converted (quantized + repackaged) from
+  `deepseek-ai/DeepSeek-V4-Flash-DSpark`.
+
+### Model Sources [optional]
+
+- **Repository:** https://github.com/RaffaelloMolinari/FaStar
+- **Paper [optional]:** N/A (engineering showcase; see the upstream DeepSeek V4 / DSpark
+  materials).
+- **Demo [optional]:** FaStar ships a built-in HTTP chat UI (`ds4_npu_engine --serve`); see the
+  repository README.
+
+## Uses
+
+### Direct Use
+
+On-device research inference of DeepSeek V4 Flash on an AMD Ryzen AI 9 365 laptop NPU, using the
+FaStar engine. Suitable for studying expert virtual memory, on-NPU MLA/FFN via IRON-generated
+MLIR-AIE kernels, and consumer-NPU execution of a 284B model. Generation modes: one-shot,
+interactive multi-turn, and a streaming web chat UI.
+
+### Downstream Use [optional]
+
+As a reference format/checkpoint for porting other DeepSeek MoE models to the `.fst` container
+and XDNA2 target, or as a substrate for experimenting with NPU kernel fusion, expert prefetch
+strategies, and speculative-decoding acceptance tuning.
+
+### Out-of-Scope Use
+
+- **Not a production serving system.** Decode throughput is ~0.05 tokens/sec (see
+  [Bias, Risks, and Limitations]); do not use for latency-sensitive or multi-user serving.
+- **Not fine-tuned for any downstream task** and not safety-aligned — outputs are raw
+  completions from the converted base model.
+- **Not for deployment on non-XDNA2 hardware.** The `.fst` container, sidecars, and bundled
+  `.xclbin` kernels target the AMD Ryzen AI NPU; they are not a drop-in replacement for the
+  upstream HuggingFace checkpoint on CPU/GPU.
+- Do not use to generate misleading, harmful, or disallowed content; the upstream DeepSeek
+  acceptable-use terms apply.
+
+## Bias, Risks, and Limitations
+
+- **Throughput: ~0.05 tokens/sec.** Decode is NPU-compute-bound per dispatch. Dispatch-count
+  reduction (op-replication packing) was proven correct on silicon but does not move tok/s; a
+  packed N-copy blob does N× the NPU compute. Reaching >1 tok/s requires IRON-level kernel
+  fusion (future work).
+- **First-token latency.** The first token of a turn takes ~2 minutes (prefill of a 43-layer,
+  284B model on this NPU); subsequent tokens stream at ~0.05 tok/s.
+- **Numerical residual.** A recurrent hidden-state drift (~±40 RMS at deep layers) is intrinsic
+  to the MXFP4 / MLA numerics of this checkpoint — it is reproduced in a pure-PyTorch HF
+  ground-truth reference and is not a FaStar bug. Output remains coherent English with greedy
+  prefill argmax matching the HF ground truth.
+- **Memory footprint.** ~150 GB of expert weights; requires 64 GB RAM and ~200 GB NVMe SSD via
+  expert paging. Expert cache thrash drops throughput if the working set exceeds RAM.
+- **Inherited biases.** All biases, risks, and limitations of the upstream DeepSeek V4 Flash
+  base model carry over unchanged — this is a lossy-format conversion, not an alignment
+  intervention.
+
+### Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations
+inherited from the upstream DeepSeek model, the very low decode throughput, and the on-device
+NPU-only target. Verify the DeepSeek license terms before redistribution or commercial use.
+
+## How to Get Started with the Model
+
+Requirements: AMD Ryzen AI 9 365 (XDNA2 NPU), 64 GB RAM, ~200 GB NVMe SSD, Ubuntu 24.04 with
+XRT, AIEBU, and the AMDXDNA driver.
+
+```bash
+# Build the FaStar engine (CMake fetches header-only deps on first configure)
+git clone https://github.com/RaffaelloMolinari/FaStar.git
+cd FaStar
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
+cmake --build build -j$(nproc)
+
+# XRT lives in /usr on Ubuntu, not /opt/xilinx
+export XILINX_XRT=/usr
+
+# One-shot generation (greedy, deterministic):
+./build/ds4_npu_engine --model deepseek_v4_dspark.fst \
+                       --prompt "Explain quantum computing" --tokens 128 --temp 0.0
+
+# Speculative decoding with the DSpark draft model:
+./build/ds4_npu_engine --model deepseek_v4_dspark.fst --draft_model dspark_draft.fst \
+                       --prompt "Hello world" --tokens 256
+
+# Web UI / chat server:
+./build/ds4_npu_engine --model deepseek_v4_dspark.fst --serve --port 8080
+```
+
+This HuggingFace repo must contain: the `.fst` file, its `.fst.hc` / `.fst.norm` /
+`.fst.tid2eid` sidecars, `tokenizer.json`, and (for speculative decoding) `dspark_draft.fst`.
+The `.fst.norm` and `.fst.hc` sidecars are required for coherent output.
+
+## Training Details
+
+This model was **not trained** — it is a conversion of an existing checkpoint. Training details
+below describe the *source* model and the conversion procedure.
+
+### Training Data
+
+Inherited from the upstream `deepseek-ai/DeepSeek-V4-Flash-DSpark` checkpoint. See that model
+card for the original training data.
+
+### Training Procedure
+
+#### Preprocessing [optional]
+
+Conversion to the `.fst` container (no weight fine-tuning):
+
+```bash
+python3 scripts/fst_converter.py --model deepseek-ai/DeepSeek-V4-Flash-DSpark \
+                                 --output deepseek_v4_dspark.fst
+```
+
+The `.fst` format stores a page-aligned config header, shared tensors (attention, router, norms)
+in Q8_0 / BF16, and expert blocks as dense DS4 MXFP4 (17-byte blocks: 1 e8m0 scale + 16 FP4
+nibbles). Verify integrity with `scripts/verify_fst.py` and `scripts/check_fst.py`.
+
+#### Training Hyperparameters
+
+- **Training regime:** N/A — not trained. Inference weights are MXFP4 (experts) + Q8_0/BF16
+  (shared tensors), dequantized to BF16 on-NPU for compute.
+
+#### Speeds, Sizes, Times [optional]
+
+- **Parameters:** ~284B (MoE).
+- **Layers:** 43.
+- **Experts:** 1376 total (~32 per layer) + 1 shared expert per layer, routed with top-k = 6
+  and hash routing over the first 3 layers (route_scale = 1.5).
+- **Hidden size:** 4096; expert intermediate: 2048; vocabulary: 129,280.
+- **Attention:** MLA — Q latent 1024, KV latent 512.
+- **Decode throughput:** ~0.05 tokens/sec on Ryzen AI 9 365 NPU.
+- **Prefill latency:** ~2 minutes for the first token of a turn.
+- **Container size:** ~150 GB on disk (paged across SSD/RAM/NPU).
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+Numerical correctness was verified against a pure-PyTorch HuggingFace ground-truth reference
+(`hf_bos_ref.py`, `hf_gen_ref.py`) using the real DeepSeek weights, and qualitative coherence
+was checked with free-form prompts (e.g. *"The importance of NPU…"*).
+
+#### Factors
+
+Per-layer cosine similarity vs. the HF reference (MLA projections, FFN GEMM, dequantization);
+greedy prefill argmax token match; multi-token speculative-decoding acceptance rate; output
+coherence over a 43-layer prefill.
+
+#### Metrics
+
+- GEMM / dequant cosine similarity vs. reference: ~0.998 (MLA), ~0.998 (FFN),
+  ~0.999998 (fused FFN gate, deterministic).
+- Greedy prefill argmax: matches HF ground truth (e.g. argmax 671 = "The").
+- Speculative-decoding acceptance: ~0.5–0.67 depending on configuration.
+
+### Results
+
+The engine produces coherent English, and greedy prefill argmax matches the HF ground truth.
+A deep-layer hidden-state residual (~±40 RMS at layer 42) is reproduced identically in the
+pure-PyTorch HF reference and is intrinsic to this checkpoint's MXFP4/MLA numerics, not a
+FaStar defect.
+
+#### Summary
+
+FaStar is mathematically faithful to the HuggingFace reference for this checkpoint, at the cost
+of very low on-NPU throughput (~0.05 tok/s). Correctness holds; throughput is the open problem.
+
+## Model Examination [optional]
+
+Per-layer cosine audits of MLA (`wq_a`, `wkv_a`, `wo_a`, output projection), FFN
+(gate/up/down GEMM), and MXFP4 dequantization are built into the repository
+(`scripts/fst_layer_benchmark.py`, `scripts/verify_fst_weights.py`,
+`scripts/verify_fused_ffn.py`, plus `tools/` probes). These localize divergence to the
+intrinsic deep-layer residual noted above.
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute)
+presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). This is an
+**inference-only** artifact; the dominant carbon cost was paid upstream during DeepSeek's
+original training and is not attributable to this conversion.
+
+- **Hardware Type:** AMD Ryzen AI 9 365 laptop (XDNA2 NPU), 64 GB RAM, NVMe SSD.
+- **Hours used:** Conversion + on-device verification only (single laptop, no cluster training).
+- **Cloud Provider:** None — entirely on-device.
+- **Compute Region:** Local consumer hardware.
+- **Carbon Emitted:** Negligible for this artifact (inference/conversion on one laptop);
+  upstream training emissions belong to DeepSeek-AI.
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+Decoder-only MoE Transformer with Multi-head Latent Attention (Q latent 1024, KV latent 512),
+per-layer shared + routed experts (top-k = 6, hash-routed over the first 3 layers with
+route_scale = 1.5, swiglu_limit = 10), MXFP4 expert weights with e8m0 scales, Q8_0/BF16 shared
+tensors, and a V4 KV-compressor that streams 512-dim compressed KV rows. A small DSpark draft
+model enables speculative decoding. Objective: next-token text generation.
+
+### Compute Infrastructure
+
+#### Hardware
+
+- APU: AMD Ryzen AI 9 365 (XDNA2 NPU is the compute target; iGPU unused).
+- RAM: 64 GB minimum (~54 GB peak RSS observed).
+- Storage: NVMe SSD, ~200 GB free (experts live on SSD; read latency dominates miss cost).
+- GPU: not required.
+
+#### Software
+
+- Ubuntu 24.04; XRT (Xilinx Runtime) at `/usr`; AIEBU assembler; AMDXDNA kernel driver.
+- FaStar engine (`ds4_npu_engine`, C++17, CMake build).
+- NPU kernels generated with AMD IRON / MLIR-AIE; prebuilt `.xclbin` + `_insts.bin` ship in the
+  repo `kernels/` directory.
+- Python 3 + `tokenizers` for the HF BPE tokenizer bridge.
+
+## Citation [optional]
+
+If you use this work, please cite the upstream DeepSeek model and the FaStar engine.
+
+**BibTeX:**
+
+```bibtex
+@misc{deepseek-v4-flash-dspark,
+  author       = {{DeepSeek-AI}},
+  title        = {DeepSeek V4 Flash (DSpark)},
+  howpublished = {\url{https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark}}
+}
+
+@misc{fastar,
+  author       = {Raffaello Molinari},
+  title        = {FaStar: Expert Virtual Memory Inference Engine for DeepSeek V4 Flash on AMD Ryzen AI NPU},
+  howpublished = {\url{https://github.com/RaffaelloMolinari/FaStar}}
+}
+```
+
+**APA:**
+
+DeepSeek-AI. (n.d.). *DeepSeek V4 Flash (DSpark)*. HuggingFace. https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark
+
+Molinari, R. (n.d.). *FaStar: Expert virtual memory inference engine for DeepSeek V4 Flash on AMD Ryzen AI NPU*. GitHub. https://github.com/RaffaelloMolinari/FaStar
+
+## Glossary [optional]
+
+- **`.fst`:** FaStar's page-aligned model container (config header + shared tensors in
+  Q8_0/BF16 + dense DS4 MXFP4 expert blocks).
+- **Expert virtual memory:** MoE experts treated as virtual-memory pages — stored on SSD,
+  cached in RAM (`ExpertPager` LRU), uploaded to NPU scratch on demand.
+- **MXFP4:** Microscaling FP4 — 16 FP4 nibbles per block plus one shared e8m0 scale (17 bytes
+  per expert block).
+- **MLA:** Multi-head Latent Attention — compresses Q (1024-d) and KV (512-d) into latent
+  projections; the KV cache stores compressed latents + positional embeddings.
+- **DSpark:** Draft-model speculative decoding — a small model proposes tokens that the main
+  model verifies in a block.
+- **IRON / MLIR-AIE:** AMD's compiler toolchain that generates the NPU kernels (`.xclbin`).
+- **hw_context:** An AMDXDNA device context bound to an xclbin; the driver caps simultaneous
+  contexts at 9 (managed by `AiebuKernelCache`).
+
+## More Information [optional]
+
+See the [FaStar repository README](https://github.com/RaffaelloMolinari/FaStar) for the full
+architecture overview, CLI flags, web-UI API, kernel rebuild instructions, and known
+limitations.
+
+## Model Card Authors [optional]
+
+Raffaello Molinari (`.fst` conversion + model card). Model weights and architecture by
+DeepSeek-AI.
+
+## Model Card Contact
+
+Open an issue on the [FaStar GitHub repository](https://github.com/RaffaelloMolinari/FaStar).
+````