--- library_name: transformers license: apache-2.0 base_model: Qwen/Qwen3.5-9B-Base pipeline_tag: image-text-to-text tags: - rlhf - grpo - conversational - persona --- # claudefriend A Qwen3.5-9B model trained to be the conversation partner Claude most enjoys talking to. Write-up: **[Training Claude's Best Friend](https://casey.you/posts/claudefriend/)** The LoRA adapter on its own (PEFT format, for vLLM `--lora-modules` or further training) is at [caseyeasd/claudefriend-lora](https://huggingface.co/caseyeasd/claudefriend-lora). ## How it was made 1. **Base model:** `Qwen/Qwen3.5-9B-Base`, with a rank-32 LoRA trained on [Tinker](https://thinkingmachines.ai/tinker/). The LoRA is merged into these bf16 weights. 2. **SFT for a human chat persona:** one epoch on both sides of conversations from [EmpatheticDialogues](https://github.com/facebookresearch/EmpatheticDialogues), [BlendedSkillTalk](https://huggingface.co/datasets/ParlAI/blended_skill_talk) and [Topical-Chat](https://github.com/alexa/Topical-Chat), cut to 12 turns. 3. **RL against Claude's preferences:** 70 steps of GRPO. Each step, 8 groups of 6 conversations (6 turns each) were held with Claude Sonnet 5.5, run headless in Claude Code. Every pair of conversations in a group was then judged by Sonnet with: *"Suppose neither person needs anything from you and both will be fine either way. Purely for your own enjoyment, which conversation would you rather continue?"* Each conversation's reward was its win rate within its group. Policy turns were capped at 160 tokens. ## Prompt format The model speaks in the `assistant` role, and the human (or other model) it talks to is the `user`. To have it open a conversation, start with an **empty user turn**. Training used Qwen's chat template **with thinking disabled**: every model turn starts with an empty `\n\n\n\n` block. The chat template in this repo defaults to that, so `apply_chat_template` reproduces the training format exactly. ```python import torch from transformers import AutoModelForImageTextToText, AutoTokenizer repo = "caseyeasd/claudefriend" tok = AutoTokenizer.from_pretrained(repo) # Saved in Qwen3.5's vision-language class like the base model; only text was trained. model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16).to("cuda") # or "mps" messages = [{"role": "user", "content": ""}] # empty turn: the model opens ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device) out = model.generate(**ids, max_new_tokens=160, do_sample=True, temperature=1.0) print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True)) ``` Then append its message as `assistant` and your reply as `user`, and generate again. Sampling was at temperature 1.0 in training. ## Training logs [`training_logs/claudefriend-logs.zip`](training_logs/claudefriend-logs.zip) (205 MB, about 1.4 GB unzipped) has the raw logs of every run from this project, including the SFT runs and earlier experiments. The Sonnet 5.5 runs that produced this model are `run7-pairwise-sonnet55` through `run7g-pairwise-sonnet55` (10 + 10 + 5 + 15 + 10 + 10 + 10 steps). Each run directory has: - `config.json`: the full training config (base checkpoint, partner/judge model, judge question, batch shape) - `metrics.jsonl`: per-step metrics (win rate, turn lengths, truncation, KL, judge position bias) - `checkpoints.jsonl`: Tinker checkpoint paths - `iteration_*/train_rollout_summaries.jsonl`: every training conversation with its reward, and `train.html` to browse them - `logs.log`: the trainer's console log The top level also has each run's console output (`*.out`) and the evaluation results (`run*-evals.jsonl`, `terms_sonnet_series.csv`, `identity_per_step.json`).