Qwen3.8 0.6B Distill

Inspired by Empero AI's models, this model aims to bring high-quality inference onto compute constrained devices. The student model, Qwen3 0.6B, has undergone parameter-efficient distillation using traces from a Qwen3.8 27B teacher. Keep in mind that the model has been trained in an environment with low compute capabilities, and you should mostly treat it as a proof of concept.

Hyperparameters used for training

Dataset Steps - 484
Effective Batch Size - 8
Sequence Length - 2048
Total Tokens Seen - ~4.2 million
Learning Rate - 8e-5
Learning Rate Scheduler - Linear
Warmup Steps - 15
LoRA Rank - 32
LoRA Alpha - 64
Target Modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Repo Contents

File	Size and Role
model.safetensors	1.19GB, contains model weights stored in 16-bit precision
tokenizer.json	11.4MB, tokenizer
tokenizer_config.json	10.2KB, contains specific chat template tokens
generation_config.json	236 bytes, contains default hyperparameters (like temperature and top-K)
config.json	1.6KB crucial info about the model
chat_template.jinja	4.91KB, chat template