--- library_name: peft base_model: Qwen/Qwen3-0.6B pipeline_tag: text-generation license: apache-2.0 tags: - qwen3 - lora - peft - cpt - sft - dpo - grpo - reasoning - unsloth - trl --- # finetuning-checkpoints — Qwen3-0.6B post-training pipeline Checkpoints from an end-to-end post-training run on **Qwen/Qwen3-0.6B** (~600M parameters) covering **continued pre-training (CPT) → supervised fine-tuning (SFT) → DPO → reasoning-format warm-up → GRPO**. This is a **checkpoint repository**, not a single deployable model. Pick the subfolder for the stage you want. The last stage (`grpo_qwen06_merged`) is the "final" model; `grpo_qwen06/llm_best_0` is the best GRPO adapter by eval reward. Produced by [NeelContractor/finetuning_recipes](https://github.com/NeelContractor/finetuning_recipes), based on the [avbiswas/finetuning_recipes](https://github.com/avbiswas/finetuning_recipes) course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4. --- ## Pipeline ``` ┌──────────────► dpo_qwen06 (DPO, side branch) │ Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO ──► final │ └──► DPO ``` Two branches leave SFT: preference optimization (DPO) and the reasoning branch (format warm-up, then GRPO). GRPO is trained from `reasoning_warmup_qwen06_merged`, not from the DPO adapter. --- ## What's in the repo | Path | Type | Stage | Base / notes | |---|---|---|---| | `models/cpt_qwen06/final` + `checkpoint-20…131` | LoRA adapter | CPT on arXiv paper text | `unsloth/Qwen3-0.6B-unsloth-bnb-4bit` | | `models/cpt_qwen06_merged` | Full model (fp) | merged CPT | Qwen3-0.6B | | `models/sft_qwen06/final` + `checkpoint-100/150/198` | LoRA adapter | Instruction SFT | `cpt_qwen06_merged` | | `models/sft_qwen06_merged` | Full model (fp) | merged SFT | Qwen3-0.6B | | `models/dpo_qwen06/final` + `checkpoint-50/77` | LoRA adapter (+ `ref/`) | DPO preference tuning | `sft_qwen06_merged` | | `models/reasoning_warmup_qwen06/final` + `checkpoint-250/300/324` | LoRA adapter | Reasoning-format SFT (`think`/answer) | `sft_qwen06_merged` | | `models/reasoning_warmup_qwen06_merged` | Full model (fp) | merged warm-up | Qwen3-0.6B | | `models/grpo_qwen06/llm` | LoRA adapter | GRPO (latest) | `reasoning_warmup_qwen06_merged` | | `models/grpo_qwen06/llm_best_0` | LoRA adapter | GRPO (best eval reward) | `reasoning_warmup_qwen06_merged` | | `models/grpo_qwen06_merged` | Full model (fp) | **final merged model** | Qwen3-0.6B | | `models/grpo_qwen06/logs` | JSON/JSONL | GRPO train + eval logs | — | Notes on naming: - `final/` — last adapter saved for the stage. - `checkpoint-N/` — resumable checkpoints (adapter + optimizer/scheduler/RNG state). - `*_merged/` — LoRA merged into the base, saved as a full-precision `Qwen3ForCausalLM` (safe to load directly with `transformers`). --- ## Quick start ### Load the final merged model (recommended) ```python from transformers import AutoModelForCausalLM, AutoTokenizer REPO = "NeelContractor/finetuning-checkpoints" SUB = "models/grpo_qwen06_merged" tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB) model = AutoModelForCausalLM.from_pretrained( REPO, subfolder=SUB, torch_dtype="auto", device_map="auto" ) ``` ### Load a LoRA adapter The adapter configs were written on Kaggle and reference local paths (`/kaggle/working/...`), so load the matching merged base explicitly: ```python from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer REPO = "NeelContractor/finetuning-checkpoints" base = AutoModelForCausalLM.from_pretrained( REPO, subfolder="models/reasoning_warmup_qwen06_merged", torch_dtype="auto", device_map="auto", ) model = PeftModel.from_pretrained( base, REPO, subfolder="models/grpo_qwen06/llm_best_0" ) tokenizer = AutoTokenizer.from_pretrained( REPO, subfolder="models/grpo_qwen06/llm_best_0" ) ``` ### Generate The reasoning branch expects a system prompt that asks for a `think` block: ```python SYSTEM = ( "You are a helpful, respectful and honest assistant. You are an expert in " "AI, deep learning, and machine learning research. Generate your reasoning " "first inside and tags. After , generate only " "the requested final response. When JSON is requested, the content after " " must contain only that format, without Markdown fences." ) messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": "Explain LoRA in two sentences."}, ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = tokenizer(text, return_tensors="pt").to(model.device) output = model.generate( **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9 ) print(tokenizer.decode( output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True )) ``` --- ## Training details | | | |---|---| | Base model | `Qwen/Qwen3-0.6B` (CPT loaded via `unsloth/Qwen3-0.6B-unsloth-bnb-4bit`) | | Method | LoRA (PEFT) with Unsloth kernels / TRL trainers | | LoRA rank | `r=32`, `lora_alpha=32` (CPT/SFT/DPO/warm-up), `lora_alpha=64` (GRPO) | | Target modules | `q, k, v, o, gate, up, down` projections | | LoRA dropout | 0.0 | | Hardware | Kaggle, 2× Tesla T4 (15 GB), DDP | | Precision | fp16 training on T4; merged checkpoints saved as bf16 | | Max sequence length | 1024 (SFT / reasoning / GRPO) | | Reasoning warm-up | 10,000 sampled → 6,911 after the 1024-token filter; 3 epochs, 324 steps, ~46 min on 2×T4 | | GRPO | 1 epoch, 8 train events (384 optimizer steps), ~50 min on 2×T4 | ### Data | Stage | Data | |---|---| | CPT | arXiv paper text prepared by `data_prep/cpt` | | SFT | [`paperbd/paper_instructions_300K-v1`](https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1) (default config, ~20k rows) | | DPO | Preference pairs produced/ranked by the preference-optimization scripts | | Reasoning warm-up | `reasoning` config of `paperbd/paper_instructions_300K-v1` | | GRPO | Train slice of the same `reasoning` config, scored by [`paperbd/neuraltxt-reward-22M`](https://huggingface.co/paperbd/neuraltxt-reward-22M) plus format/schema/length terms | --- ## GRPO evaluation (recorded run) From `models/grpo_qwen06/logs/eval_summary_5.json` (64 held-out prompts): | Metric | Mean | |---|---| | `total_reward` | **2.355** | | `neuraltxt_reward` | **0.519** | | `length_penalty_reward` | −0.078 | | `num_tokens` | 190.6 | | `think_format_reward` | **0.953** | | `output_datatype_reward` | 0.461 | | `output_schema_reward` | 0.461 | Datatype and schema match rate: **93.8%** each. Best inference score: **2.355**. An earlier run used a reasoning warm-up of only 3 optimizer steps, so the model never emitted `think...` blocks and every format reward stayed at zero. The current checkpoints come from the dedicated `reasoning` stage (10,000 rows, 3 epochs), which fixed it: 95.3% of eval responses contain a valid `think...` block and 93.8% match the requested datatype/schema. Semantic quality is still modest — this is a 0.6B research model, not a strong reasoner. --- ## Reproducing / extending The orchestrator builds the data, runs each stage, and optionally pushes checkpoints to the Hub: ```bash python cloud_train.py --install --num-gpus 2 \ --push-to-hub --hub-repo NeelContractor/finetuning-checkpoints ``` Run only the stages you need: ```bash # Retrain the reasoning-format warm-up, then GRPO python cloud_train.py --stages reasoning grpo --num-gpus 2 ``` See the [project README](https://github.com/NeelContractor/finetuning_recipes) for stage-by-stage details, dataset prep, and evaluation scripts. The Kaggle runbook for the reasoning branch is [REASON_TRAINING.md](https://github.com/NeelContractor/finetuning_recipes/blob/main/REASON_TRAINING.md). --- ## Limitations & intended use - Research / educational checkpoints from a short, small-model run. - Not instruction-tuned for production, not safety-tuned, not evaluated for harmful content. - Format compliance is now high (95.3% valid think blocks, 93.8% structural match), but semantic reasoning quality is limited by the 0.6B model. - Derived from Qwen3-0.6B (Apache-2.0); check the licenses of the training datasets and the reward model before redistributing or commercial use.