DeepSeek-V4.1-Flash-WSP-SFT-Compressed (7.37 GiB)

A structurally compressed, weight-space optimized, and supervised fine-tuned (SFT) edition of deepseek-ai/DeepSeek-V4.1-Flash, reducing the original 552B / 475.24 GiB checkpoint down to 7.37 GiB (model0-mp1.safetensors) with a portable pure-PyTorch CPU/GPU inference engine.

Highlights

1. Causal Encoder-Decoder (CED) Structural Compression (40 $\rightarrow$ 8 Layers):
   • Preserves the core encoder-decoder anchor and KV/Index source layers ([0, 1, 2, 8, 20, 28, 36, 39]) of the DeepSeek-V4.1-Flash CED backbone.
2. Domain-Guided MoE Expert Distillation (384 $\rightarrow$ 24 Routed Experts per Layer):
   • Ranks all 384 routed experts per layer using the sqrtsoftplus router against token embeddings of software engineering, system design, network security, CEH, penetration testing, and bug bounty curricula.
   • Retains the top 24 domain-specialized experts per layer (4 active per token) and folds unselected experts' routing affinity into their nearest retained expert.
   • Applies low-rank SVD principal subspace enhancement to attn.wo_a.weight.
3. Multi-Epoch Supervised Fine-Tuning (SFT):
   • Fine-tuned for 8 epochs (AdamW, Cosine Annealing LR, LoRA rank 16) across:
     • Software Engineering & Concurrency: Lock-safe & overflow-safe C/Rust/Python concurrent structures.
     • Distributed System Design: Atomic Redis Lua sliding-window rate limiters & idempotent architectures.
     • Network Engineering & CEH: Layer-2 DAI & DHCP Snooping, Layer-3 BGP RPKI/ROA validation, Layer-4 TCP SYN Cookies, TLS 1.3 1-RTT PFS, and Active Directory (Kerberoasting, Pass-the-Hash, Golden Ticket) detection & hardening.
     • Ethical Penetration Testing & Bug Bounty Auditing: SSRF (CWE-918) DNS-rebinding prevention, TOCTOU single-packet race condition locking (CWE-362), JWT/OAuth verification, BOLA/IDOR tenant scoping, XXE hardening, and GraphQL query complexity limits.
4. Portable Pure-PyTorch CPU/GPU Kernels:
   • Replaces Hopper/Blackwell-only tilelang kernels with a portable kernel.py so the model runs on standard CPUs and consumer GPUs.

SFT Training Progression

Epoch	Cross-Entropy Loss	Perplexity
1	14.1267	1,365,029.08
2	10.4454	34,386.74
3	8.2925	3,993.97
4	7.3276	1,521.73
5	6.7505	854.49
6	6.3308	561.59
7	6.1397	463.90
8	6.0728	433.87

Quickstart

pip install torch safetensors tokenizers
python run_deepseek_v41.py \
  --prompt "Explain how to prevent SSRF and TOCTOU race conditions in Python web APIs." \
  --max-new-tokens 256