Buckets:
| # Play 2048 with GRPO | |
| Train a language model to play 2048 with GRPO. The game runs in the OpenSpiel environment, and TRL's `GRPOTrainer` drives it through `environment_factory`: the model plays by calling a `move` tool, and the game score is the reward. | |
| The recipe follows TRL's [`examples/grpo_2048`](https://github.com/huggingface/trl/tree/main/examples/grpo_2048), which implements the game in plain Python. Here the game logic lives in an OpenEnv server instead. | |
| > [!TIP] | |
| > Unsloth maintains its own [2048 tutorial with OpenEnv](https://github.com/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_%2820B%29_Reinforcement_Learning_2048_Game.ipynb), which trains `gpt-oss-20b` with Unsloth on the same OpenSpiel server to write a 2048 strategy instead of playing move by move. | |
| > [!NOTE] | |
| > **Difficulty**: Advanced | **GPU Required**: Yes | |
| ## 1. Install dependencies | |
| ```python | |
| !pip install -Uq "trl[peft]" trackio open_spiel | |
| !pip install -q "openenv-openspiel-env @ git+https://github.com/huggingface/OpenEnv.git#subdirectory=envs/openspiel_env" | |
| ``` | |
| `open_spiel` ships the game engine (wheels for Linux and macOS). `openenv-openspiel-env` installs the client (`OpenSpielEnv`) and the server. | |
| ## 2. Start the 2048 server | |
| An OpenSpiel server serves one game, chosen with the `OPENSPIEL_GAME` variable when it starts. Run one locally for 2048: | |
| ```python | |
| import os, subprocess, sys, time | |
| import requests | |
| server = subprocess.Popen( | |
| [sys.executable, "-m", "uvicorn", "openspiel_env.server.app:app", "--port", "8000"], | |
| env={**os.environ, "OPENSPIEL_GAME": "2048"}, | |
| ) | |
| for _ in range(60): | |
| try: | |
| if requests.get("http://localhost:8000/health").ok: | |
| break | |
| except requests.ConnectionError: | |
| time.sleep(1) | |
| ``` | |
| The hosted [`openenv/openspiel_env`](https://huggingface.co/spaces/openenv/openspiel_env) Space serves Catch. To host 2048 instead, duplicate it, set the `OPENSPIEL_GAME=2048` variable in the Space settings, and use its `https://<user>-openspiel-env.hf.space` URL as `base_url` below. | |
| The server accepts 8 concurrent sessions by default (`MAX_CONCURRENT_ENVS`), one per rollout in a generation batch with the settings below. | |
| ## 3. Define the environment class | |
| `environment_factory` takes a class the trainer instantiates once per rollout. Its public methods with docstrings become tools, `reset()` returns the start of the conversation, and `get_reward()` scores the finished rollout. | |
| OpenSpiel's 2048 observation is the board as 16 tile values (`info_state`), the step reward is the points gained by merging tiles, and the actions are `0` (up), `1` (right), `2` (down) and `3` (left). | |
| ```python | |
| from openspiel_env import OpenSpielAction, OpenSpielEnv | |
| PROMPT = "Play 2048 on a 4x4 board. Use the tool `move` with one of: up, down, left, right. Maximize the score." | |
| MOVES = {"up": 0, "right": 1, "down": 2, "left": 3} | |
| class Game2048Env: | |
| def __init__(self): | |
| self.client = OpenSpielEnv(base_url="http://localhost:8000").sync() | |
| def reset(self, **kwargs) -> str: | |
| result = self.client.reset() | |
| self.observation = result.observation | |
| self.score = 0.0 | |
| self.done = result.done | |
| return f"{PROMPT}\n\n{self._render()}" | |
| def move(self, direction: str) -> str: | |
| """ | |
| Play one move in 2048. | |
| Args: | |
| direction: One of "up", "down", "left", "right". | |
| Returns: | |
| The board and score after the move. | |
| """ | |
| if self.done: | |
| raise ValueError("Game over.") | |
| action_id = MOVES.get(direction.strip().lower()) | |
| if action_id not in self.observation.legal_actions: | |
| raise ValueError(f"Illegal move: {direction}.") | |
| result = self.client.step(OpenSpielAction(action_id=action_id, game_name="2048")) | |
| self.observation = result.observation | |
| self.score += result.reward or 0.0 | |
| self.done = result.done | |
| return self._render() | |
| def get_reward(self) -> float: | |
| return self.score | |
| def _render(self) -> str: | |
| tiles = [int(v) for v in self.observation.info_state] | |
| rows = [" ".join(f"{v:4d}" for v in tiles[i : i + 4]) for i in range(0, 16, 4)] | |
| return f"score={self.score}\n" + "\n".join(rows) + f"\ndone={self.done}" | |
| ``` | |
| An error raised inside a tool is sent back to the model as the tool response, so an illegal move costs a turn without ending the game. | |
| ## 4. Train with GRPO | |
| The environment generates the starting board and owns the reward, so the trainer needs no dataset and no reward function. `max_steps` sets the length of the run. | |
| ```python | |
| from peft import LoraConfig | |
| from trl import GRPOConfig, GRPOTrainer | |
| trainer = GRPOTrainer( | |
| model="Qwen/Qwen3-4B", | |
| args=GRPOConfig( | |
| max_steps=200, | |
| chat_template_kwargs={"enable_thinking": False}, | |
| logging_steps=1, | |
| log_completions=True, | |
| num_completions_to_print=2, | |
| report_to="trackio", | |
| trackio_space_id="openenv-2048", | |
| max_completion_length=2048, | |
| per_device_train_batch_size=4, | |
| gradient_accumulation_steps=2, | |
| ), | |
| environment_factory=Game2048Env, | |
| peft_config=LoraConfig(), | |
| ) | |
| trainer.train() | |
| ``` | |
| Each generation batch holds `per_device_train_batch_size × gradient_accumulation_steps = 8` rollouts, all from the same starting prompt (`num_generations` defaults to 8). The trainer keeps one `Game2048Env` instance per rollout and reuses them across batches. A rollout ends when the model stops calling `move` or reaches `max_completion_length`, so longer completions let the model play longer games. | |
| In the Trackio dashboard, `reward` (also logged as `rewards/Game2048Env/mean`) is the mean game score and `tools/call_frequency` is how often the model calls `move`. Expect the call frequency to rise first, then the score. | |
| ## 5. Save the model | |
| ```python | |
| trainer.push_to_hub() | |
| server.terminate() | |
| ``` | |
| ## Next steps | |
| - The [end-to-end walkthrough](end-to-end-walkthrough) covers the same `environment_factory` pattern on a single-step task, and [Wordle GRPO](wordle-grpo) on another multi-turn game. | |
| - The [Training overview](../guides/training) lists the other frameworks that train on OpenEnv environments. | |
Xet Storage Details
- Size:
- 6.21 kB
- Xet hash:
- 49afdb233d3c27f3b6862bb124776e53316fa6e1949a2a6fbef990c7ad6386d5
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.