Akahsizrr/mini-deepseek-v4-flash

Mini DeepSeek V4 Flash (Fuse-2 Trained)

Overview

Fuse-2 model: Qwen3-4B host + 260 DeepSeek V4 Flash coding experts with trained bridges, routers, and repair layers.

Training Details

• Steps: 1000
• Dataset: tatsu-lab/alpaca + synthetic coding instructions
• Method: Distillation (KL divergence to Qwen3-4B teacher) + LM loss
• Learning rates: bridge_in/router/repair_down=1e-2, bridge_out/repair_up=1e-6
• NaN count: 0 (fully stable)
• GPU: A100 80GB

Trained Components

Component	Init std	Final std	Growth
bridge_in	0.020	0.053-0.091	2.6-4.5x
router	0.010	0.031-0.063	3.1-6.3x
repair_down	0.020	0.041-0.084	2.0-4.2x
bridge_out	0.0001	0.0001	(tiny, preserves generation)
repair_up	0.0001	0.0001	(tiny, preserves generation)

Usage

Load the base model from Akahsizrr/Mini-Whale-Flash/merged-v2-full, then load trainable_params.pt with strict=False:

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/Mini-Whale-Flash/merged-v2-full",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
ckpt = torch.load("trainable_params.pt", map_location="auto", weights_only=True)
ckpt_bf16 = {k: v.to(torch.bfloat16) for k, v in ckpt.items()}
model.load_state_dict(ckpt_bf16, strict=False)

Files

• trainable_params.pt — final checkpoint (step 1000)
• checkpoints/v2_step{200,400,600,800,1000}/trainable_params.pt — intermediate checkpoints