Valhari14
/

Learning-in-Interaction

@@ -1,166 +1,60 @@
 ---
-license: mit
 library_name: transformers
-base_model: meta-llama/Llama-3.1-8B-Instruct
 tags:
-- llama
-- lora
-- sft
-- alignment
 - reinforcement-learning
-- peft
-- transformers
-- archer
 ---
-# Learning in Interaction using ArCHer
-> Interactive fine-tuning and alignment experiments using LoRA, supervised learning, and reinforcement-learning-based adaptation on LLaMA-3.1-8B-Instruct.
----
-## Overview
-This repository contains checkpoints, training configurations, and experimental artifacts for the project:
-**Learning in Interaction using ArCHer**
-The goal of this work is to explore interactive learning paradigms for large language models through:
-- Supervised Fine-Tuning (SFT)
-- LoRA-based parameter-efficient adaptation
-- Preference and reward-based optimization
-- Future GRPO / ArCHer reinforcement learning methods
-The experiments are built on top of Meta's **LLaMA-3.1-8B-Instruct** model using the Hugging Face ecosystem.
----
-## Base Model
-- **Model:** `meta-llama/Llama-3.1-8B-Instruct`
-- **Framework:** Transformers + PEFT
-- **Training Infrastructure:** Hugging Face Trainer
-- **Fine-Tuning Method:** LoRA
----
-## Repository Structure
-```text
-Llama-3.1-8B-Instruct/
-│
-├── sft_lora/
-│   └── checkpoint-1611/
-│       ├── adapter_model.safetensors
-│       ├── adapter_config.json
-│       ├── tokenizer.json
-│       ├── tokenizer_config.json
-│       ├── trainer_state.json
-│       ├── training_args.bin
-│       ├── optimizer.pt
-│       ├── scheduler.pt
-│       ├── rng_state.pth
-│       └── chat_template.jinja
-│
-├── sft_lora_with_val_splits/
-│   └── checkpoint-2016/
-│       ├── adapter_model.safetensors
-│       ├── adapter_config.json
-│       ├── tokenizer.json
-│       ├── tokenizer_config.json
-│       ├── trainer_state.json
-│       ├── training_args.bin
-│       ├── optimizer.pt
-│       ├── scheduler.pt
-│       ├── rng_state.pth
-│       └── chat_template.jinja
-```
----
-## Experiment Summary
-| Experiment | Checkpoint | Description |
-|---|---|---|
-| SFT + LoRA | `checkpoint-1611` | Initial supervised fine-tuning using LoRA adapters |
-| SFT + LoRA + Validation Splits | `checkpoint-2016` | Validation-aware training with improved monitoring |
-| ArCHer / GRPO (Upcoming) | TBD | Reinforcement-learning-based interactive optimization |
----
-## Loading a Checkpoint
-### Using Transformers + PEFT
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from peft import PeftModel
-base_model = "meta-llama/Llama-3.1-8B-Instruct"
-# Load base model
-model = AutoModelForCausalLM.from_pretrained(
-    base_model,
-    device_map="auto",
-    torch_dtype="auto"
-)
-# Load LoRA adapter
-model = PeftModel.from_pretrained(
-    model,
-    "path_to_checkpoint"
-)
-# Load tokenizer
-tokenizer = AutoTokenizer.from_pretrained(base_model)
-```
----
-## Example Inference
-```python
-inputs = tokenizer(
-    "Explain reinforcement learning in simple terms.",
-    return_tensors="pt"
-).to(model.device)
-outputs = model.generate(
-    **inputs,
-    max_new_tokens=128
-)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
----
-## Notes
-- Only selected checkpoints and milestone runs are stored.
-- Large files are managed using Git LFS.
-- Intermediate checkpoints are omitted to reduce repository size.
-- Additional experiments and evaluation results will be added over time.
----
-## Planned Work
-Future extensions include:
-- GRPO-based reinforcement learning
-- Human feedback integration
-- Multi-turn interaction benchmarks
-- Automated best-checkpoint selection
-- Evaluation pipelines and benchmarking suites
----
-## License & Usage
-This repository follows the licensing terms of the underlying base model:
-- `meta-llama/Llama-3.1-8B-Instruct`
-Users must comply with the original Llama license and usage restrictions when using derived checkpoints.

 ---
 library_name: transformers
+base_model: Qwen/Qwen3.5-9B
 tags:
+- qwen3.5
+- playpen
+- score
+- self-correction
 - reinforcement-learning
+- dialogue-games
 ---
+# Learning in Interaction
+This repository contains the merged model checkpoints from our study of **SCoRe (Self-Correction via Reinforcement Learning)** in Playpen dialogue games.
+We investigate whether a two-stage self-correction RL procedure, originally developed for mathematical and coding tasks, transfers to multi-turn dialogue games with game-based feedback.
+## Models
+| Checkpoint | Description |
+|---|---|
+| `qwen-sft-sp_merged` | Qwen3.5-9B after SFT training |
+| `score-Qwen3.5-9B-singleplayer_merged-15_08` | SCoRe trained from the base Qwen3.5-9B |
+| `score-Qwen3.5-9B-sft-init-sp_merged-16_08` | SFT warm-up followed by SCoRe on Qwen3.5-9B |
+## Training
+- **Base model:** Qwen3.5-9B
+- **Environment:** Playpen dialogue games
+- **Games:** AdventureGame, TextMapWorld, TextMapWorld GraphReasoning, TextMapWorld SpecificRoom, Wordle
+- **SFT:** 2,677 successful episodes, 3 epochs, LoRA
+- **SCoRe Stage I:** 3 epochs
+- **SCoRe Stage II:** 5 epochs
+- **Hardware:** 2 × NVIDIA H100 80GB
+- **Total training time:** approximately 7 days, excluding hyperparameter tuning
+## Evaluation
+We evaluate:
+- win rate across the first and second attempts
+- abort rate
+- self-correction and regression rates
+- turns and token usage
+- Playpen/ClemScore
+- single-player and static-store evaluations
+The current experiments are exploratory, with **10 evaluation episodes per game and model**. Results should therefore be interpreted as directional rather than conclusive.
+## Intended Use
+These checkpoints are intended for research on **reinforcement learning, self-correction, and learning from interaction in dialogue games**. They have not been evaluated as general-purpose improved versions of Qwen3.5-9B.
+## Acknowledgements
+This work builds on **SCoRe**, **Playpen**, and **Clembench**, and uses Qwen3.5-9B as the base model.