| --- |
| library_name: pytorch |
| tags: |
| - reinforcement-learning |
| - atari |
| - phoenix |
| - ppo |
| - custom-implementation |
| model-index: |
| - name: ppo-ALE/Phoenix-v5 |
| results: |
| - task: |
| type: reinforcement-learning |
| name: reinforcement-learning |
| dataset: |
| name: ALE/Phoenix-v5 |
| type: ALE/Phoenix-v5 |
| metrics: |
| - type: mean_reward |
| value: 4687.19 +/- 974.80 |
| name: mean_reward |
| --- |
| # PPO agent playing ALE/Phoenix-v5 |
|
|
| A from-scratch PyTorch implementation (PPO) trained for 2,000,000 agent steps (8,000,000 frames). |
|
|
|  |
| <video src="replay.mp4" controls></video> |
|
|
| ## Evaluation |
| | metric | value | |
| |---|---| |
| | mean score (stochastic policy) | 4687.2 ± 974.8 over 32 episodes | |
| | mean score (greedy policy) | 4578.8 | |
| | environment | `ALE/Phoenix-v5` (sticky actions 0.25, sign-clipped rewards for training, raw score for evaluation) | |
|
|
| ## Model |
| * Input: 4 stacked 84x84 grayscale frames. Network: `impala` CNN trunk (15 conv layers) + 2-layer actor and critic heads; 1,223,129 parameters. |
| * Actions: 8 discrete (NOOP, FIRE, RIGHT, LEFT, DOWN, RIGHTFIRE, LEFTFIRE, DOWNFIRE). |
|
|
| ## Hyper-parameters |
| ```json |
| { |
| "algo": "ppo", |
| "game": "phoenix", |
| "seed": 0, |
| "tag": "", |
| "total_steps": 2000000, |
| "num_envs": 16, |
| "sticky": 0.25, |
| "noop_max": 30, |
| "max_frames": 108000, |
| "clip_reward": true, |
| "arch": "impala", |
| "hidden": 256, |
| "lr": 0.00025, |
| "lr_anneal": true, |
| "adam_eps": 1e-05, |
| "max_grad_norm": 0.5, |
| "gamma": 0.99, |
| "gae_lambda": 0.95, |
| "n_steps": 128, |
| "ent_coef": 0.01, |
| "vf_coef": 0.5, |
| "norm_adv": true, |
| "clip_eps": 0.1, |
| "ppo_epochs": 4, |
| "minibatches": 4, |
| "clip_vloss": false, |
| "target_kl": null, |
| "baseline": "batch", |
| "max_episode_steps": 4000, |
| "chunk": 512, |
| "eval_every": 300000, |
| "eval_episodes": 8, |
| "ckpt_every": 500000, |
| "print_every_s": 20.0, |
| "device": "auto" |
| } |
| ``` |
|
|
| See `load_example.py` for loading. Weights are stored as `model.safetensors`. |
|
|
