Add model card / README for the checkpoint repository

README.md

@@ -0,0 +1,222 @@
+---
+library_name: peft
+base_model: Qwen/Qwen3-0.6B
+pipeline_tag: text-generation
+license: apache-2.0
+tags:
+  - qwen3
+  - lora
+  - peft
+  - cpt
+  - sft
+  - dpo
+  - grpo
+  - reasoning
+  - unsloth
+  - trl
+---
+
+# finetuning-checkpoints — Qwen3-0.6B post-training pipeline
+
+Checkpoints from an end-to-end post-training run on **Qwen/Qwen3-0.6B** (~600M
+parameters) covering **continued pre-training (CPT) → supervised fine-tuning
+(SFT) → DPO → reasoning-format warm-up → GRPO**.
+
+This is a **checkpoint repository**, not a single deployable model. Pick the
+subfolder for the stage you want. The last stage (`grpo_qwen06_merged`) is the
+"final" model; `grpo_qwen06/llm_best_0` is the best GRPO adapter by eval reward.
+
+Produced by [NeelContractor/finetuning_recipes](https://github.com/NeelContractor/finetuning_recipes),
+based on the [avbiswas/finetuning_recipes](https://github.com/avbiswas/finetuning_recipes)
+course repo. Built with Unsloth, TRL, and PEFT on Kaggle 2× Tesla T4.
+
+---
+
+## Pipeline
+
+```
+                 ┌──────────────► dpo_qwen06 (DPO, side branch)
+                 │
+Qwen3-0.6B ──► CPT ──► SFT ──┬──► reasoning_warmup ──► GRPO  ──► final
+                             │
+                             └──► DPO
+```
+
+Two branches leave SFT: preference optimization (DPO) and the reasoning branch
+(format warm-up, then GRPO). GRPO is trained from `reasoning_warmup_qwen06_merged`,
+not from the DPO adapter.
+
+---
+
+## What's in the repo
+
+| Path | Type | Stage | Base / notes |
+|---|---|---|---|
+| `models/cpt_qwen06/final` + `checkpoint-20…131` | LoRA adapter | CPT on arXiv paper text | `unsloth/Qwen3-0.6B-unsloth-bnb-4bit` |
+| `models/cpt_qwen06_merged` | Full model (fp) | merged CPT | Qwen3-0.6B |
+| `models/sft_qwen06/final` + `checkpoint-100/150/198` | LoRA adapter | Instruction SFT | `cpt_qwen06_merged` |
+| `models/sft_qwen06_merged` | Full model (fp) | merged SFT | Qwen3-0.6B |
+| `models/dpo_qwen06/final` + `checkpoint-50/77` | LoRA adapter (+ `ref/`) | DPO preference tuning | `sft_qwen06_merged` |
+| `models/reasoning_warmup_qwen06/final` + `checkpoint-3` | LoRA adapter | Reasoning-format SFT (`think`/answer) | `sft_qwen06_merged` |
+| `models/reasoning_warmup_qwen06_merged` | Full model (fp) | merged warm-up | Qwen3-0.6B |
+| `models/grpo_qwen06/llm` | LoRA adapter | GRPO (latest) | `reasoning_warmup_qwen06_merged` |
+| `models/grpo_qwen06/llm_best_0` | LoRA adapter | GRPO (best eval reward) | `reasoning_warmup_qwen06_merged` |
+| `models/grpo_qwen06_merged` | Full model (fp) | **final merged model** | Qwen3-0.6B |
+| `models/grpo_qwen06/logs` | JSON/JSONL | GRPO train + eval logs | — |
+
+Notes on naming:
+- `final/` — last adapter saved for the stage.
+- `checkpoint-N/` — resumable checkpoints (adapter + optimizer/scheduler/RNG state).
+- `*_merged/` — LoRA merged into the base, saved as a full-precision
+  `Qwen3ForCausalLM` (safe to load directly with `transformers`).
+
+---
+
+## Quick start
+
+### Load the final merged model (recommended)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+REPO = "NeelContractor/finetuning-checkpoints"
+SUB = "models/grpo_qwen06_merged"
+
+tokenizer = AutoTokenizer.from_pretrained(REPO, subfolder=SUB)
+model = AutoModelForCausalLM.from_pretrained(
+    REPO, subfolder=SUB, torch_dtype="auto", device_map="auto"
+)
+```
+
+### Load a LoRA adapter
+
+The adapter configs were written on Kaggle and reference local paths
+(`/kaggle/working/...`), so load the matching merged base explicitly:
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+REPO = "NeelContractor/finetuning-checkpoints"
+
+base = AutoModelForCausalLM.from_pretrained(
+    REPO, subfolder="models/reasoning_warmup_qwen06_merged",
+    torch_dtype="auto", device_map="auto",
+)
+model = PeftModel.from_pretrained(
+    base, REPO, subfolder="models/grpo_qwen06/llm_best_0"
+)
+tokenizer = AutoTokenizer.from_pretrained(
+    REPO, subfolder="models/grpo_qwen06/llm_best_0"
+)
+```
+
+### Generate
+
+The reasoning branch expects a system prompt that asks for a `think` block:
+
+```python
+SYSTEM = (
+    "You are a helpful, respectful and honest assistant. You are an expert in "
+    "AI, deep learning, and machine learning research. Generate your reasoning "
+    "first inside  thinking and </think> tags. After </think>, generate only "
+    "the requested final response. When JSON is requested, the content after "
+    "</think> must contain only that format, without Markdown fences."
+)
+
+messages = [
+    {"role": "system", "content": SYSTEM},
+    {"role": "user", "content": "Explain LoRA in two sentences."},
+]
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+output = model.generate(
+    **inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9
+)
+print(tokenizer.decode(
+    output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
+))
+```
+
+---
+
+## Training details
+
+| | |
+|---|---|
+| Base model | `Qwen/Qwen3-0.6B` (CPT loaded via `unsloth/Qwen3-0.6B-unsloth-bnb-4bit`) |
+| Method | LoRA (PEFT) with Unsloth kernels / TRL trainers |
+| LoRA rank | `r=32`, `lora_alpha=32` (CPT/SFT/DPO/warm-up), `lora_alpha=64` (GRPO) |
+| Target modules | `q, k, v, o, gate, up, down` projections |
+| LoRA dropout | 0.0 |
+| Hardware | Kaggle, 2× Tesla T4 (15 GB), DDP |
+| Precision | fp16 training on T4; merged checkpoints saved as bf16 |
+| Max sequence length | 1024 (SFT / reasoning / GRPO) |
+
+### Data
+
+| Stage | Data |
+|---|---|
+| CPT | arXiv paper text prepared by `data_prep/cpt` |
+| SFT | [`paperbd/paper_instructions_300K-v1`](https://huggingface.co/datasets/paperbd/paper_instructions_300K-v1) (default config, ~20k rows) |
+| DPO | Preference pairs produced/ranked by the preference-optimization scripts |
+| Reasoning warm-up | `reasoning` config of `paperbd/paper_instructions_300K-v1` |
+| GRPO | Train slice of the same `reasoning` config, scored by [`paperbd/neuraltxt-reward-22M`](https://huggingface.co/paperbd/neuraltxt-reward-22M) plus format/schema/length terms |
+
+---
+
+## GRPO evaluation (recorded run)
+
+From `models/grpo_qwen06/logs/eval_summary_5.json` (64 held-out prompts):
+
+| Metric | Mean |
+|---|---|
+| `total_reward` | **0.440** |
+| `neuraltxt_reward` | **0.486** |
+| `length_penalty_reward` | −0.094 |
+| `num_tokens` | 101 |
+| `think_format_reward` | 0.000 |
+| `output_datatype_reward` | 0.000 |
+| `output_schema_reward` | 0.000 |
+
+**Known limitation:** this run's reasoning warm-up was only 3 optimizer steps, so
+the model never learned to emit `think...` blocks and the format-related rewards
+stayed at zero. The pipeline has since added a dedicated `reasoning` stage that
+trains the format on ~10k rows for 3 epochs; the artifacts in this repo predate
+that change. Do not treat these checkpoints as strong reasoning models — they
+are research/teaching artifacts from a short, small-model run.
+
+---
+
+## Reproducing / extending
+
+The orchestrator builds the data, runs each stage, and optionally pushes
+checkpoints to the Hub:
+
+```bash
+python cloud_train.py --install --num-gpus 2 \
+  --push-to-hub --hub-repo NeelContractor/finetuning-checkpoints
+```
+
+Run only the stages you need:
+
+```bash
+# Retrain the reasoning-format warm-up, then GRPO
+python cloud_train.py --stages reasoning grpo --num-gpus 2
+```
+
+See the [project README](https://github.com/NeelContractor/finetuning_recipes)
+for stage-by-stage details, dataset prep, and evaluation scripts.
+
+---
+
+## Limitations & intended use
+
+- Research / educational checkpoints from a short, small-model run.
+- Not instruction-tuned for production, not safety-tuned, not evaluated for
+  harmful content.
+- Reasoning format is unreliable in these artifacts (see metrics above).
+- Derived from Qwen3-0.6B (Apache-2.0); check the licenses of the training
+  datasets and the reward model before redistributing or commercial use.