card: replace placeholder

6a9ecac verified about 1 month ago

2.63 kB

---
license: apache-2.0
base_model: Qwen/Qwen3-4B
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- grpo
- text-to-sql
- hill-climb
- while-ai
datasets:
- while-ai/text-to-sql-shop
---
# text-to-sql-shop-r2

<!-- while-ai: where this fits -->
*Recipe: [recipes/04-train/text-to-sql](https://github.com/whilehq/whileai-sdk/tree/main/recipes/04-train/text-to-sql) · Collection: [Analyst](https://huggingface.co/collections/while-ai/analyst-6aada4c396dfe31da497862d)*

**One rung of the text-to-SQL hill climb on Qwen3-4B, thinking on, graded by executing the query.** Round `r2` starts from `r1`: GRPO 200 steps, lr 5e-5, beta 0.01. The eval rows for this round are the `eval-r2` config of the [text-to-sql-shop](https://huggingface.co/datasets/while-ai/text-to-sql-shop) dataset.

## Where it sits in the climb

81 held-out tasks, 4 samples each, temperature 0.7. The row in bold is this adapter.

| Round | From | Method | pass@1 (95% CI) | pass^4 | has_sql |
|---|---|---|---|---|---|
| r1 | base | GRPO 100 steps, lr 2e-5, beta 0.04, HF generate | 0.58 (0.52..0.64) | 0.29 | 0.89 |
| **r2** | r1 | GRPO 200 steps, lr 5e-5, beta 0.01 | 0.60 (0.54..0.67) | 0.34 | 0.90 |
| sft-think | base | self-distillation: 199 verified traces, hosted SFT 2 epochs | 0.60 (0.54..0.67) | 0.39 | 0.96 |
| r3 | r2 | GRPO 1,000 steps, lr 2e-5, beta 0.01, vLLM generation, 8 prompts per generate | 0.61 (0.56..0.67) | 0.29 | 0.86 |
| r4 | r3 | GRPO 1,000 more steps, same settings | 0.61 (0.54..0.67) | 0.32 | 0.85 |
Rounds r1 to r4 are within each other's intervals: two thousand GRPO steps did not move pass@1 on this holdout. The move came in r5, from a re-cut prompt band and masked truncation, and is the `text-to-sql-shop-r5` repo. The recipe README explains what each round changed and why.
## Arms in this repo
The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. `checkpoints/` never ships.
| folder | arm |
|---|---|
| `.` | round `r2` adapter |
## Load
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
model = PeftModel.from_pretrained(base, "while-ai/text-to-sql-shop-r2")  # the headline arm
```
## Reproduce
```bash
git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/04-train/text-to-sql
modal run train_grpo_modal.py --run-name t2s-r1
```
The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.

	---
	license: apache-2.0
	base_model: Qwen/Qwen3-4B
	library_name: peft
	pipeline_tag: text-generation
	tags:
	- lora
	- grpo
	- text-to-sql
	- hill-climb
	- while-ai
	datasets:
	- while-ai/text-to-sql-shop
	---

	# text-to-sql-shop-r2

	<!-- while-ai: where this fits -->
	Recipe: [recipes/04-train/text-to-sql](https://github.com/whilehq/whileai-sdk/tree/main/recipes/04-train/text-to-sql) · Collection: [Analyst](https://huggingface.co/collections/while-ai/analyst-6aada4c396dfe31da497862d)

	One rung of the text-to-SQL hill climb on Qwen3-4B, thinking on, graded by executing the query. Round `r2` starts from `r1`: GRPO 200 steps, lr 5e-5, beta 0.01. The eval rows for this round are the `eval-r2` config of the [text-to-sql-shop](https://huggingface.co/datasets/while-ai/text-to-sql-shop) dataset.

	## Where it sits in the climb

	81 held-out tasks, 4 samples each, temperature 0.7. The row in bold is this adapter.

	\| Round \| From \| Method \| pass@1 (95% CI) \| pass^4 \| has_sql \|
	\|---\|---\|---\|---\|---\|---\|
	\| r1 \| base \| GRPO 100 steps, lr 2e-5, beta 0.04, HF generate \| 0.58 (0.52..0.64) \| 0.29 \| 0.89 \|
	\| r2 \| r1 \| GRPO 200 steps, lr 5e-5, beta 0.01 \| 0.60 (0.54..0.67) \| 0.34 \| 0.90 \|
	\| sft-think \| base \| self-distillation: 199 verified traces, hosted SFT 2 epochs \| 0.60 (0.54..0.67) \| 0.39 \| 0.96 \|
	\| r3 \| r2 \| GRPO 1,000 steps, lr 2e-5, beta 0.01, vLLM generation, 8 prompts per generate \| 0.61 (0.56..0.67) \| 0.29 \| 0.86 \|
	\| r4 \| r3 \| GRPO 1,000 more steps, same settings \| 0.61 (0.54..0.67) \| 0.32 \| 0.85 \|

	Rounds r1 to r4 are within each other's intervals: two thousand GRPO steps did not move pass@1 on this holdout. The move came in r5, from a re-cut prompt band and masked truncation, and is the `text-to-sql-shop-r5` repo. The recipe README explains what each round changed and why.

	## Arms in this repo

	The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. `checkpoints/` never ships.

	\| folder \| arm \|
	\|---\|---\|
	\| `.` \| round `r2` adapter \|

	## Load

	```python
	from peft import PeftModel
	from transformers import AutoModelForCausalLM

	base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
	model = PeftModel.from_pretrained(base, "while-ai/text-to-sql-shop-r2") # the headline arm
	```


	## Reproduce

	```bash
	git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/04-train/text-to-sql
	modal run train_grpo_modal.py --run-name t2s-r1
	```

	The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.