README.md

---
library_name: transformers
base_model: Qwen/Qwen3.5-9B
tags:
- qwen3.5
- playpen
- score
- self-correction
- reinforcement-learning
- dialogue-games
---

# Learning in Interaction

This repository contains the merged model checkpoints from our study of **SCoRe (Self-Correction via Reinforcement Learning)** in Playpen dialogue games.

We investigate whether a two-stage self-correction RL procedure, originally developed for mathematical and coding tasks, transfers to multi-turn dialogue games with game-based feedback.

## Models

| Checkpoint | Description |
|---|---|
| `qwen-sft-sp_merged` | Qwen3.5-9B after SFT training |
| `score-Qwen3.5-9B-singleplayer_merged-15_08` | SCoRe trained from the base Qwen3.5-9B |
| `score-Qwen3.5-9B-sft-init-sp_merged-16_08` | SFT warm-up followed by SCoRe on Qwen3.5-9B |


## Training

- **Base model:** Qwen3.5-9B
- **Environment:** Playpen dialogue games
- **Games:** AdventureGame, TextMapWorld, TextMapWorld GraphReasoning, TextMapWorld SpecificRoom, Wordle
- **SFT:** 2,677 successful episodes, 3 epochs, LoRA
- **SCoRe Stage I:** 3 epochs
- **SCoRe Stage II:** 5 epochs
- **Hardware:** 2 × NVIDIA H100 80GB
- **Total training time:** approximately 7 days, excluding hyperparameter tuning



## Evaluation

We evaluate:

- win rate across the first and second attempts
- abort rate
- self-correction and regression rates
- turns and token usage
- Playpen/ClemScore
- single-player and static-store evaluations

The current experiments are exploratory, with **10 evaluation episodes per game and model**. Results should therefore be interpreted as directional rather than conclusive.

## Intended Use

These checkpoints are intended for research on **reinforcement learning, self-correction, and learning from interaction in dialogue games**. They have not been evaluated as general-purpose improved versions of Qwen3.5-9B.

## Acknowledgements

This work builds on **SCoRe**, **Playpen**, and **Clembench**, and uses Qwen3.5-9B as the base model.