Model card: update GRPO eval with reasoning warm-up results

de36c2d verified 3 days ago

8.51 kB

---
library_name: peft
base_model: Qwen/Qwen3-0.6B
pipeline_tag: text-generation
license: apache-2.0
tags:
  - qwen3
  - lora
  - peft
  - cpt
  - sft
  - dpo
  - grpo
  - reasoning
  - unsloth
  - trl
---
# finetuning-checkpoints — Qwen3-0.6B post-training pipeline

Checkpoints from an end-to-end post-training run on **Qwen/Qwen3-0.6B** (~600M
parameters) covering **continued pre-training (CPT) → supervised fine-tuning
(SFT) → DPO → reasoning-format warm-up → GRPO**.

This is a **checkpoint repository**, not a single deployable model. Pick the
subfolder for the stage you want. The last stage (`grpo_qwen06_merged`) is the
"final" model; `grpo_qwen06/llm_best_0` is the best GRPO adapter by eval reward.

Produced by [NeelContractor/finetuning_recipes](https://github.com/NeelContractor/finetuning_recipes),
based on the [avbiswas/finetuning_recipes](https://github.com/avbiswas/finetuning_recipes)
course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4.

---

## Pipeline

```
                 ┌──────────────► dpo_qwen06 (DPO, side branch)
                 │
Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO  ──► final
                             │
                             └──► DPO
```

Two branches leave SFT: preference optimization (DPO) and the reasoning branch
(format warm-up, then GRPO). GRPO is trained from `reasoning_warmup_qwen06_merged`,
not from the DPO adapter.

---

## What's in the repo

| Path | Type | Stage | Base / notes |
|---|---|---|---|
| `models/cpt_qwen06/final` + `checkpoint-20…131` | LoRA adapter | CPT on arXiv paper text | `unsloth/Qwen3-0.6B-unsloth-bnb-4bit` |
| `models/cpt_qwen06_merged` | Full model (fp) | merged CPT | Qwen3-0.6B |
| `models/sft_qwen06/final` + `checkpoint-100/150/198` | LoRA adapter | Instruction SFT | `cpt_qwen06_merged` |
| `models/sft_qwen06_merged` | Full model (fp) | merged SFT | Qwen3-0.6B |
| `models/dpo_qwen06/final` + `checkpoint-50/77` | LoRA adapter (+ `ref/`) | DPO preference tuning | `sft_qwen06_merged` |
| `models/reasoning_warmup_qwen06/final` + `checkpoint-250/300/324` | LoRA adapter | Reasoning-format SFT (`think`/answer) | `sft_qwen06_merged` |
| `models/reasoning_warmup_qwen06_merged` | Full model (fp) | merged warm-up | Qwen3-0.6B |
| `models/grpo_qwen06/llm` | LoRA adapter | GRPO (latest) | `reasoning_warmup_qwen06_merged` |
| `models/grpo_qwen06/llm_best_0` | LoRA adapter | GRPO (best eval reward) | `reasoning_warmup_qwen06_merged` |
| `models/grpo_qwen06_merged` | Full model (fp) | **final merged model** | Qwen3-0.6B |
| `models/grpo_qwen06/logs` | JSON/JSONL | GRPO train + eval logs | — |

Notes on naming:
- `final/` — last adapter saved for the stage.
- `checkpoint-N/` — resumable checkpoints (adapter + optimizer/scheduler/RNG state).
- `*_merged/` — LoRA merged into the base, saved as a full-precision
  `Qwen3ForCausalLM` (safe to load directly with `transformers`).

---

## Quick start

### Load the final merged model (recommended)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "NeelContractor/finetuning-checkpoints"
SUB = "models/grpo_qwen06_merged"
tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
model = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder=SUB, torch_dtype="auto", device_map="auto"
)
```

### Load a LoRA adapter

The adapter configs were written on Kaggle and reference local paths
(`/kaggle/working/...`), so load the matching merged base explicitly:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "NeelContractor/finetuning-checkpoints"
base = AutoModelForCausalLM.from_pretrained(
    REPO, subfolder="models/reasoning_warmup_qwen06_merged",
    torch_dtype="auto", device_map="auto",
)
model = PeftModel.from_pretrained(
    base, REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
tokenizer = AutoTokenizer.from_pretrained(
    REPO, subfolder="models/grpo_qwen06/llm_best_0"
)
```

### Generate

The reasoning branch expects a system prompt that asks for a `think` block:

```python
SYSTEM = (
    "You are a helpful, respectful and honest assistant. You are an expert in "
    "AI, deep learning, and machine learning research. Generate your reasoning "
    "first inside <think> and </think> tags. After </think>, generate only "
    "the requested final response. When JSON is requested, the content after "
    "</think> must contain only that format, without Markdown fences."
)
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Explain LoRA in two sentences."},
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9
)
print(tokenizer.decode(
    output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
))
```

---

## Training details

| | |
|---|---|
| Base model | `Qwen/Qwen3-0.6B` (CPT loaded via `unsloth/Qwen3-0.6B-unsloth-bnb-4bit`) |
| Method | LoRA (PEFT) with Unsloth kernels / TRL trainers |
| LoRA rank | `r=32`, `lora_alpha=32` (CPT/SFT/DPO/warm-up), `lora_alpha=64` (GRPO) |
| Target modules | `q, k, v, o, gate, up, down` projections |
| LoRA dropout | 0.0 |
| Hardware | Kaggle, 2× Tesla T4 (15 GB), DDP |
| Precision | fp16 training on T4; merged checkpoints saved as bf16 |
| Max sequence length | 1024 (SFT / reasoning / GRPO) |
| Reasoning warm-up | 10,000 sampled → 6,911 after the 1024-token filter; 3 epochs, 324 steps, ~46 min on 2×T4 |
| GRPO | 1 epoch, 8 train events (384 optimizer steps), ~50 min on 2×T4 |

### Data

| Stage | Data |
|---|---|
| CPT | arXiv paper text prepared by `data_prep/cpt` |
| SFT | [`paperbd/paper_instructions_300K-v1`](https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1) (default config, ~20k rows) |
| DPO | Preference pairs produced/ranked by the preference-optimization scripts |
| Reasoning warm-up | `reasoning` config of `paperbd/paper_instructions_300K-v1` |
| GRPO | Train slice of the same `reasoning` config, scored by [`paperbd/neuraltxt-reward-22M`](https://huggingface.co/paperbd/neuraltxt-reward-22M) plus format/schema/length terms |

---

## GRPO evaluation (recorded run)

From `models/grpo_qwen06/logs/eval_summary_5.json` (64 held-out prompts):

| Metric | Mean |
|---|---|
| `total_reward` | **2.355** |
| `neuraltxt_reward` | **0.519** |
| `length_penalty_reward` | −0.078 |
| `num_tokens` | 190.6 |
| `think_format_reward` | **0.953** |
| `output_datatype_reward` | 0.461 |
| `output_schema_reward` | 0.461 |

Datatype and schema match rate: **93.8%** each. Best inference score: **2.355**.

An earlier run used a reasoning warm-up of only 3 optimizer steps, so the model
never emitted `think...` blocks and every format reward stayed at zero. The
current checkpoints come from the dedicated `reasoning` stage (10,000 rows,
3 epochs), which fixed it: 95.3% of eval responses contain a valid
`think...` block and 93.8% match the requested datatype/schema. Semantic
quality is still modest — this is a 0.6B research model, not a strong reasoner.

---

## Reproducing / extending

The orchestrator builds the data, runs each stage, and optionally pushes
checkpoints to the Hub:

```bash
python cloud_train.py --install --num-gpus 2 \
  --push-to-hub --hub-repo NeelContractor/finetuning-checkpoints
```

Run only the stages you need:

```bash
# Retrain the reasoning-format warm-up, then GRPO
python cloud_train.py --stages reasoning grpo --num-gpus 2
```

See the [project README](https://github.com/NeelContractor/finetuning_recipes)
for stage-by-stage details, dataset prep, and evaluation scripts. The
Kaggle runbook for the reasoning branch is
[REASON_TRAINING.md](https://github.com/NeelContractor/finetuning_recipes/blob/main/REASON_TRAINING.md).

---

## Limitations & intended use

- Research / educational checkpoints from a short, small-model run.
- Not instruction-tuned for production, not safety-tuned, not evaluated for
  harmful content.
- Format compliance is now high (95.3% valid think blocks, 93.8% structural
  match), but semantic reasoning quality is limited by the 0.6B model.
- Derived from Qwen3-0.6B (Apache-2.0); check the licenses of the training
  datasets and the reward model before redistributing or commercial use.

	---
	library_name: peft
	base_model: Qwen/Qwen3-0.6B
	pipeline_tag: text-generation
	license: apache-2.0
	tags:
	- qwen3
	- lora
	- peft
	- cpt
	- sft
	- dpo
	- grpo
	- reasoning
	- unsloth
	- trl
	---

	# finetuning-checkpoints — Qwen3-0.6B post-training pipeline

	Checkpoints from an end-to-end post-training run on Qwen/Qwen3-0.6B (~600M
	parameters) covering **continued pre-training (CPT) → supervised fine-tuning
	(SFT) → DPO → reasoning-format warm-up → GRPO**.

	This is a checkpoint repository, not a single deployable model. Pick the
	subfolder for the stage you want. The last stage (`grpo_qwen06_merged`) is the
	"final" model; `grpo_qwen06/llm_best_0` is the best GRPO adapter by eval reward.

	Produced by [NeelContractor/finetuning_recipes](https://github.com/NeelContractor/finetuning_recipes),
	based on the [avbiswas/finetuning_recipes](https://github.com/avbiswas/finetuning_recipes)
	course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4.

	---

	## Pipeline

	```
	┌──────────────► dpo_qwen06 (DPO, side branch)
	│
	Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO ──► final
	│
	└──► DPO
	```

	Two branches leave SFT: preference optimization (DPO) and the reasoning branch
	(format warm-up, then GRPO). GRPO is trained from `reasoning_warmup_qwen06_merged`,
	not from the DPO adapter.

	---

	## What's in the repo

	\| Path \| Type \| Stage \| Base / notes \|
	\|---\|---\|---\|---\|
	\| `models/cpt_qwen06/final` + `checkpoint-20…131` \| LoRA adapter \| CPT on arXiv paper text \| `unsloth/Qwen3-0.6B-unsloth-bnb-4bit` \|
	\| `models/cpt_qwen06_merged` \| Full model (fp) \| merged CPT \| Qwen3-0.6B \|
	\| `models/sft_qwen06/final` + `checkpoint-100/150/198` \| LoRA adapter \| Instruction SFT \| `cpt_qwen06_merged` \|
	\| `models/sft_qwen06_merged` \| Full model (fp) \| merged SFT \| Qwen3-0.6B \|
	\| `models/dpo_qwen06/final` + `checkpoint-50/77` \| LoRA adapter (+ `ref/`) \| DPO preference tuning \| `sft_qwen06_merged` \|
	\| `models/reasoning_warmup_qwen06/final` + `checkpoint-250/300/324` \| LoRA adapter \| Reasoning-format SFT (`think`/answer) \| `sft_qwen06_merged` \|
	\| `models/reasoning_warmup_qwen06_merged` \| Full model (fp) \| merged warm-up \| Qwen3-0.6B \|
	\| `models/grpo_qwen06/llm` \| LoRA adapter \| GRPO (latest) \| `reasoning_warmup_qwen06_merged` \|
	\| `models/grpo_qwen06/llm_best_0` \| LoRA adapter \| GRPO (best eval reward) \| `reasoning_warmup_qwen06_merged` \|
	\| `models/grpo_qwen06_merged` \| Full model (fp) \| final merged model \| Qwen3-0.6B \|
	\| `models/grpo_qwen06/logs` \| JSON/JSONL \| GRPO train + eval logs \| — \|

	Notes on naming:
	- `final/` — last adapter saved for the stage.
	- `checkpoint-N/` — resumable checkpoints (adapter + optimizer/scheduler/RNG state).
	- `*_merged/` — LoRA merged into the base, saved as a full-precision
	`Qwen3ForCausalLM` (safe to load directly with `transformers`).

	---

	## Quick start

	### Load the final merged model (recommended)

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	REPO = "NeelContractor/finetuning-checkpoints"
	SUB = "models/grpo_qwen06_merged"

	tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
	model = AutoModelForCausalLM.from_pretrained(
	REPO, subfolder=SUB, torch_dtype="auto", device_map="auto"
	)
	```

	### Load a LoRA adapter

	The adapter configs were written on Kaggle and reference local paths
	(`/kaggle/working/...`), so load the matching merged base explicitly:

	```python
	from peft import PeftModel
	from transformers import AutoModelForCausalLM, AutoTokenizer

	REPO = "NeelContractor/finetuning-checkpoints"

	base = AutoModelForCausalLM.from_pretrained(
	REPO, subfolder="models/reasoning_warmup_qwen06_merged",
	torch_dtype="auto", device_map="auto",
	)
	model = PeftModel.from_pretrained(
	base, REPO, subfolder="models/grpo_qwen06/llm_best_0"
	)
	tokenizer = AutoTokenizer.from_pretrained(
	REPO, subfolder="models/grpo_qwen06/llm_best_0"
	)
	```

	### Generate

	The reasoning branch expects a system prompt that asks for a `think` block:

	```python
	SYSTEM = (
	"You are a helpful, respectful and honest assistant. You are an expert in "
	"AI, deep learning, and machine learning research. Generate your reasoning "
	"first inside <think> and </think> tags. After </think>, generate only "
	"the requested final response. When JSON is requested, the content after "
	"</think> must contain only that format, without Markdown fences."
	)

	messages = [
	{"role": "system", "content": SYSTEM},
	{"role": "user", "content": "Explain LoRA in two sentences."},
	]
	text = tokenizer.apply_chat_template(
	messages, tokenize=False, add_generation_prompt=True
	)
	inputs = tokenizer(text, return_tensors="pt").to(model.device)
	output = model.generate(
	**inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9
	)
	print(tokenizer.decode(
	output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
	))
	```

	---

	## Training details

	\| \| \|
	\|---\|---\|
	\| Base model \| `Qwen/Qwen3-0.6B` (CPT loaded via `unsloth/Qwen3-0.6B-unsloth-bnb-4bit`) \|
	\| Method \| LoRA (PEFT) with Unsloth kernels / TRL trainers \|
	\| LoRA rank \| `r=32`, `lora_alpha=32` (CPT/SFT/DPO/warm-up), `lora_alpha=64` (GRPO) \|
	\| Target modules \| `q, k, v, o, gate, up, down` projections \|
	\| LoRA dropout \| 0.0 \|
	\| Hardware \| Kaggle, 2× Tesla T4 (15 GB), DDP \|
	\| Precision \| fp16 training on T4; merged checkpoints saved as bf16 \|
	\| Max sequence length \| 1024 (SFT / reasoning / GRPO) \|
	\| Reasoning warm-up \| 10,000 sampled → 6,911 after the 1024-token filter; 3 epochs, 324 steps, ~46 min on 2×T4 \|
	\| GRPO \| 1 epoch, 8 train events (384 optimizer steps), ~50 min on 2×T4 \|

	### Data

	\| Stage \| Data \|
	\|---\|---\|
	\| CPT \| arXiv paper text prepared by `data_prep/cpt` \|
	\| SFT \| [`paperbd/paper_instructions_300K-v1`](https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1) (default config, ~20k rows) \|
	\| DPO \| Preference pairs produced/ranked by the preference-optimization scripts \|
	\| Reasoning warm-up \| `reasoning` config of `paperbd/paper_instructions_300K-v1` \|
	\| GRPO \| Train slice of the same `reasoning` config, scored by [`paperbd/neuraltxt-reward-22M`](https://huggingface.co/paperbd/neuraltxt-reward-22M) plus format/schema/length terms \|

	---

	## GRPO evaluation (recorded run)

	From `models/grpo_qwen06/logs/eval_summary_5.json` (64 held-out prompts):

	\| Metric \| Mean \|
	\|---\|---\|
	\| `total_reward` \| 2.355 \|
	\| `neuraltxt_reward` \| 0.519 \|
	\| `length_penalty_reward` \| −0.078 \|
	\| `num_tokens` \| 190.6 \|
	\| `think_format_reward` \| 0.953 \|
	\| `output_datatype_reward` \| 0.461 \|
	\| `output_schema_reward` \| 0.461 \|

	Datatype and schema match rate: 93.8% each. Best inference score: 2.355.

	An earlier run used a reasoning warm-up of only 3 optimizer steps, so the model
	never emitted `think...` blocks and every format reward stayed at zero. The
	current checkpoints come from the dedicated `reasoning` stage (10,000 rows,
	3 epochs), which fixed it: 95.3% of eval responses contain a valid
	`think...` block and 93.8% match the requested datatype/schema. Semantic
	quality is still modest — this is a 0.6B research model, not a strong reasoner.

	---

	## Reproducing / extending

	The orchestrator builds the data, runs each stage, and optionally pushes
	checkpoints to the Hub:

	```bash
	python cloud_train.py --install --num-gpus 2 \
	--push-to-hub --hub-repo NeelContractor/finetuning-checkpoints
	```

	Run only the stages you need:

	```bash
	# Retrain the reasoning-format warm-up, then GRPO
	python cloud_train.py --stages reasoning grpo --num-gpus 2
	```

	See the [project README](https://github.com/NeelContractor/finetuning_recipes)
	for stage-by-stage details, dataset prep, and evaluation scripts. The
	Kaggle runbook for the reasoning branch is
	[REASON_TRAINING.md](https://github.com/NeelContractor/finetuning_recipes/blob/main/REASON_TRAINING.md).

	---

	## Limitations & intended use

	- Research / educational checkpoints from a short, small-model run.
	- Not instruction-tuned for production, not safety-tuned, not evaluated for
	harmful content.
	- Format compliance is now high (95.3% valid think blocks, 93.8% structural
	match), but semantic reasoning quality is limited by the 0.6B model.
	- Derived from Qwen3-0.6B (Apache-2.0); check the licenses of the training
	datasets and the reward model before redistributing or commercial use.