Blame for README.md

---
library_name: peft
base_model: Qwen/Qwen3-0.6B
pipeline_tag: text-generation
license: apache-2.0
tags:
  - qwen3
  - lora
  - peft
  - cpt
  - sft
  - dpo
  - grpo
  - reasoning
  - unsloth
  - trl
---

# finetuning-checkpoints — Qwen3-0.6B post-training pipeline

Checkpoints from an end-to-end post-training run on **Qwen/Qwen3-0.6B** (~600M
parameters) covering **continued pre-training (CPT) → supervised fine-tuning
(SFT) → DPO → reasoning-format warm-up → GRPO**.

This is a **checkpoint repository**, not a single deployable model. Pick the
subfolder for the stage you want. The last stage (`grpo_qwen06_merged`) is the
"final" model; `grpo_qwen06/llm_best_0` is the best GRPO adapter by eval reward.

Produced by [NeelContractor/finetuning_recipes](https://github.com/NeelContractor/finetuning_recipes),
based on the [avbiswas/finetuning_recipes](https://github.com/avbiswas/finetuning_recipes)
course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4.

---

## Pipeline

```
                 ┌──────────────► dpo_qwen06 (DPO, side branch)
                 │
Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO  ──► final
                             │
                             └──► DPO
```

Two branches leave SFT: preference optimization (DPO) and the reasoning branch
(format warm-up, then GRPO). GRPO is trained from `reasoning_warmup_qwen06_merged`,
not from the DPO adapter.

---

## What's in the repo

| Path | Type | Stage | Base / notes |
|---|---|---|---|
| `models/cpt_qwen06/final` + `checkpoint-20…131` | LoRA adapter | CPT on arXiv paper text | `unsloth/Qwen3-0.6B-unsloth-bnb-4bit` |
| `models/cpt_qwen06_merged` | Full model (fp) | merged CPT | Qwen3-0.6B |
| `models/sft_qwen06/final` + `checkpoint-100/150/198` | LoRA adapter | Instruction SFT | `cpt_qwen06_merged` |
| `models/sft_qwen06_merged` | Full model (fp) | merged SFT | Qwen3-0.6B |
| `models/dpo_qwen06/final` + `checkpoint-50/77` | LoRA adapter (+ `ref/`) | DPO preference tuning | `sft_qwen06_merged` |
| `models/reasoning_warmup_qwen06/final` + `checkpoint-250/300/324` | LoRA adapter | Reasoning-format SFT (`think`/answer) | `sft_qwen06_merged` |
| `models/reasoning_warmup_qwen06_merged` | Full model (fp) | merged warm-up | Qwen3-0.6B |
| `models/grpo_qwen06/llm` | LoRA adapter | GRPO (latest) | `reasoning_warmup_qwen06_merged` |
| `models/grpo_qwen06/llm_best_0` | LoRA adapter | GRPO (best eval reward) | `reasoning_warmup_qwen06_merged` |
| `models/grpo_qwen06_merged` | Full model (fp) | **final merged model** | Qwen3-0.6B |
| `models/grpo_qwen06/logs` | JSON/JSONL | GRPO train + eval logs | — |

Notes on naming:
- `final/` — last adapter saved for the stage.
- `checkpoint-N/` — resumable checkpoints (adapter + optimizer/scheduler/RNG state).
- `*_merged/` — LoRA merged into the base, saved as a full-precision
  `Qwen3ForCausalLM` (safe to load directly with `transformers`).

---

## Quick start

### Load the final merged model (recommended)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "NeelContractor/finetuning-checkpoints"
SUB = "models/grpo_qwen06_merged"

tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
model = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder=SUB, torch_dtype="auto", device_map="auto"
)
```

### Load a LoRA adapter

The adapter configs were written on Kaggle and reference local paths
(`/kaggle/working/...`), so load the matching merged base explicitly:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "NeelContractor/finetuning-checkpoints"

base = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="models/reasoning_warmup_qwen06_merged",
    torch_dtype="auto", device_map="auto",
)
model = PeftModel.from_pretrained(
    base, REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
tokenizer = AutoTokenizer.from_pretrained(
    REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
```

### Generate

The reasoning branch expects a system prompt that asks for a `think` block:

```python
SYSTEM = (
    "You are a helpful, respectful and honest assistant. You are an expert in "
    "AI, deep learning, and machine learning research. Generate your reasoning "
    "first inside <think> and </think> tags. After </think>, generate only "
    "the requested final response. When JSON is requested, the content after "
    "</think> must contain only that format, without Markdown fences."
)

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Explain LoRA in two sentences."},
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9
)
print(tokenizer.decode(
    output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
))
```

---

## Training details

| | |
|---|---|
| Base model | `Qwen/Qwen3-0.6B` (CPT loaded via `unsloth/Qwen3-0.6B-unsloth-bnb-4bit`) |
| Method | LoRA (PEFT) with Unsloth kernels / TRL trainers |
| LoRA rank | `r=32`, `lora_alpha=32` (CPT/SFT/DPO/warm-up), `lora_alpha=64` (GRPO) |
| Target modules | `q, k, v, o, gate, up, down` projections |
| LoRA dropout | 0.0 |
| Hardware | Kaggle, 2× Tesla T4 (15 GB), DDP |
| Precision | fp16 training on T4; merged checkpoints saved as bf16 |
| Max sequence length | 1024 (SFT / reasoning / GRPO) |
| Reasoning warm-up | 10,000 sampled → 6,911 after the 1024-token filter; 3 epochs, 324 steps, ~46 min on 2×T4 |
| GRPO | 1 epoch, 8 train events (384 optimizer steps), ~50 min on 2×T4 |

### Data

| Stage | Data |
|---|---|
| CPT | arXiv paper text prepared by `data_prep/cpt` |
| SFT | [`paperbd/paper_instructions_300K-v1`](https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1) (default config, ~20k rows) |
| DPO | Preference pairs produced/ranked by the preference-optimization scripts |
| Reasoning warm-up | `reasoning` config of `paperbd/paper_instructions_300K-v1` |
| GRPO | Train slice of the same `reasoning` config, scored by [`paperbd/neuraltxt-reward-22M`](https://huggingface.co/paperbd/neuraltxt-reward-22M) plus format/schema/length terms |

---

## GRPO evaluation (recorded run)

From `models/grpo_qwen06/logs/eval_summary_5.json` (64 held-out prompts):

| Metric | Mean |
|---|---|
| `total_reward` | **2.355** |
| `neuraltxt_reward` | **0.519** |
| `length_penalty_reward` | −0.078 |
| `num_tokens` | 190.6 |
| `think_format_reward` | **0.953** |
| `output_datatype_reward` | 0.461 |
| `output_schema_reward` | 0.461 |

Datatype and schema match rate: **93.8%** each. Best inference score: **2.355**.

An earlier run used a reasoning warm-up of only 3 optimizer steps, so the model
never emitted `think...` blocks and every format reward stayed at zero. The
current checkpoints come from the dedicated `reasoning` stage (10,000 rows,
3 epochs), which fixed it: 95.3% of eval responses contain a valid
`think...` block and 93.8% match the requested datatype/schema. Semantic
quality is still modest — this is a 0.6B research model, not a strong reasoner.

---

## Reproducing / extending

The orchestrator builds the data, runs each stage, and optionally pushes
checkpoints to the Hub:

```bash
python cloud_train.py --install --num-gpus 2 \
  --push-to-hub --hub-repo NeelContractor/finetuning-checkpoints
```

Run only the stages you need:

```bash
# Retrain the reasoning-format warm-up, then GRPO
python cloud_train.py --stages reasoning grpo --num-gpus 2
```

See the [project README](https://github.com/NeelContractor/finetuning_recipes)
for stage-by-stage details, dataset prep, and evaluation scripts. The
Kaggle runbook for the reasoning branch is
[REASON_TRAINING.md](https://github.com/NeelContractor/finetuning_recipes/blob/main/REASON_TRAINING.md).

---

## Limitations & intended use

- Research / educational checkpoints from a short, small-model run.
- Not instruction-tuned for production, not safety-tuned, not evaluated for
  harmful content.
- Format compliance is now high (95.3% valid think blocks, 93.8% structural
  match), but semantic reasoning quality is limited by the 0.6B model.
- Derived from Qwen3-0.6B (Apache-2.0); check the licenses of the training
  datasets and the reward model before redistributing or commercial use.