SFT cold-start LoRA adapter for Qwen3.5-9B (r=32, 32/32 layer coverage)

c6bedb9 verified 3 months ago

4.22 kB

---
base_model: Qwen/Qwen3.5-9B
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen3.5-9B
- lora
- peft
- sft
- trl
---
# sinopia-b57f6cad

A LoRA adapter on **[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)**,
trained as an SFT cold start: it distils a frozen teacher's *thoughts* into the
policy, so that a later RL stage has something better than a cold model to
refine.

`sinopia` is the red underdrawing laid down beneath a fresco — the sketch the
finished painting is built over. That is what this checkpoint is.

## The prompt contract (read this first)

This model is trained to continue from **inside an already-open `<think>`
block**. The prompt must end with the assistant header followed by `<think>\n`;
the chat template will not add that for you. Completions are shaped:

```
</think>
THOUGHT: {thought}
{action}
```

so a downstream parser can recover `(thought, action)` by splitting on the
closing `</think>` and taking the last fenced ```bash block as the action. If
you render the prompt with `apply_chat_template(..., add_generation_prompt=True)`
and stop there, the model will not behave as trained.
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen3.5-9B"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto",
    attn_implementation="flash_attention_2",
)
model = PeftModel.from_pretrained(model, "iamPi/sinopia-b57f6cad")
model = model.merge_and_unload()          # optional; needed to serve with vLLM
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
if not prompt.rstrip().endswith("<think>"):
    prompt += "<think>\n"                 # the contract above
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=1792, do_sample=True, temperature=0.8)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
```
## Adapter
| | |
|---|---|
| rank / alpha / dropout | 32 / 128 / 0.05 (α/r = 4) |
| trainable params | 80,216,064 |
| tensors | 400 |
| base | `Qwen/Qwen3.5-9B` (bf16, 8.95 B) |
### Target modules — why there are ten, not seven
Qwen3.5-9B is a **hybrid linear-attention** model: `layer_types` alternates
three `linear_attention` layers with one `full_attention`, so only **8 of 32**
layers carry `q_proj`/`k_proj`/`v_proj`/`o_proj`. The conventional LoRA target
list therefore adapts every MLP but the *mixing operator* of just a quarter of
the layers.

This adapter adds the Gated DeltaNet projections so all 32 layers are covered:

```
q_proj  k_proj  v_proj  o_proj          -> 8 full-attention layers
gate_proj  up_proj  down_proj           -> all 32 MLPs
in_proj_qkv  in_proj_z  out_proj        -> 24 linear-attention layers
```

`in_proj_a` / `in_proj_b` are deliberately excluded: they emit 32 values, so
r=32 would be full-rank on them for negligible capacity, and they gate the
recurrence's decay/beta dynamics where a perturbation destabilises more easily
than it helps. `conv1d` is depthwise, which low-rank factorisation does not fit.

## Training

| | |
|---|---|
| objective | SFT, loss on the completion only (prompt masked) |
| epochs | 1 (567 optimizer steps) |
| global batch | 32 (4×DDP × micro 2 × grad-accum 4) |
| lr / schedule | 1e-4, cosine, 3% warmup |
| precision | bf16, gradient checkpointing, flash-attention-2 |
| max length | unbounded (`padding_free`, no truncation) |

Trained on 4×A100 80GB with TRL 1.10 / transformers 5.15 / PEFT 0.20 / torch 2.13.

## Limitations

- **Not evaluated here.** This card reports how it was trained, not how well it
  scores. Treat it as a starting point for an RL stage, not a finished model.
- **One epoch.** No convergence claim is made.
- Trained on agentic shell-tool trajectories, so behaviour outside that shape —
  and outside the `<think>` contract above — is untested.

## License

The adapter carries no license of its own; it is derivative of
`Qwen/Qwen3.5-9B` and inherits that model's terms. Consult the base model card
before use.

	---
	base_model: Qwen/Qwen3.5-9B
	library_name: peft
	pipeline_tag: text-generation
	tags:
	- base_model:adapter:Qwen/Qwen3.5-9B
	- lora
	- peft
	- sft
	- trl
	---

	# sinopia-b57f6cad

	A LoRA adapter on [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B),
	trained as an SFT cold start: it distils a frozen teacher's thoughts into the
	policy, so that a later RL stage has something better than a cold model to
	refine.

	`sinopia` is the red underdrawing laid down beneath a fresco — the sketch the
	finished painting is built over. That is what this checkpoint is.

	## The prompt contract (read this first)

	This model is trained to continue from **inside an already-open `<think>`
	block**. The prompt must end with the assistant header followed by `<think>\n`;
	the chat template will not add that for you. Completions are shaped:

	```
	</think>
	THOUGHT: {thought}

	{action}
	```

	so a downstream parser can recover `(thought, action)` by splitting on the
	closing `</think>` and taking the last fenced ```bash block as the action. If
	you render the prompt with `apply_chat_template(..., add_generation_prompt=True)`
	and stop there, the model will not behave as trained.

	## Usage

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	from peft import PeftModel

	BASE = "Qwen/Qwen3.5-9B"
	tok = AutoTokenizer.from_pretrained(BASE)
	model = AutoModelForCausalLM.from_pretrained(
	BASE, dtype=torch.bfloat16, device_map="auto",
	attn_implementation="flash_attention_2",
	)
	model = PeftModel.from_pretrained(model, "iamPi/sinopia-b57f6cad")
	model = model.merge_and_unload() # optional; needed to serve with vLLM

	prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	if not prompt.rstrip().endswith("<think>"):
	prompt += "<think>\n" # the contract above

	ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
	out = model.generate(**ids, max_new_tokens=1792, do_sample=True, temperature=0.8)
	print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))
	```

	## Adapter

	\| \| \|
	\|---\|---\|
	\| rank / alpha / dropout \| 32 / 128 / 0.05 (α/r = 4) \|
	\| trainable params \| 80,216,064 \|
	\| tensors \| 400 \|
	\| base \| `Qwen/Qwen3.5-9B` (bf16, 8.95 B) \|

	### Target modules — why there are ten, not seven

	Qwen3.5-9B is a hybrid linear-attention model: `layer_types` alternates
	three `linear_attention` layers with one `full_attention`, so only 8 of 32
	layers carry `q_proj`/`k_proj`/`v_proj`/`o_proj`. The conventional LoRA target
	list therefore adapts every MLP but the mixing operator of just a quarter of
	the layers.

	This adapter adds the Gated DeltaNet projections so all 32 layers are covered:

	```
	q_proj k_proj v_proj o_proj -> 8 full-attention layers
	gate_proj up_proj down_proj -> all 32 MLPs
	in_proj_qkv in_proj_z out_proj -> 24 linear-attention layers
	```

	`in_proj_a` / `in_proj_b` are deliberately excluded: they emit 32 values, so
	r=32 would be full-rank on them for negligible capacity, and they gate the
	recurrence's decay/beta dynamics where a perturbation destabilises more easily
	than it helps. `conv1d` is depthwise, which low-rank factorisation does not fit.

	## Training

	\| \| \|
	\|---\|---\|
	\| objective \| SFT, loss on the completion only (prompt masked) \|
	\| epochs \| 1 (567 optimizer steps) \|
	\| global batch \| 32 (4×DDP × micro 2 × grad-accum 4) \|
	\| lr / schedule \| 1e-4, cosine, 3% warmup \|
	\| precision \| bf16, gradient checkpointing, flash-attention-2 \|
	\| max length \| unbounded (`padding_free`, no truncation) \|

	Trained on 4×A100 80GB with TRL 1.10 / transformers 5.15 / PEFT 0.20 / torch 2.13.

	## Limitations

	- Not evaluated here. This card reports how it was trained, not how well it
	scores. Treat it as a starting point for an RL stage, not a finished model.
	- One epoch. No convergence claim is made.
	- Trained on agentic shell-tool trajectories, so behaviour outside that shape —
	and outside the `<think>` contract above — is untested.

	## License

	The adapter carries no license of its own; it is derivative of
	`Qwen/Qwen3.5-9B` and inherits that model's terms. Consult the base model card
	before use.