Update README.md
Browse filesCenter gameplay GIFs in model README
README.md
CHANGED
|
@@ -1,3 +1,157 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- reinforcement-learning
|
| 5 |
+
- q-learning
|
| 6 |
+
- frozenlake
|
| 7 |
+
- gymnasium
|
| 8 |
+
- tabular-q-learning
|
| 9 |
+
model-index:
|
| 10 |
+
- name: FrozenLake-Q-Learning-Ducky
|
| 11 |
+
results:
|
| 12 |
+
- task:
|
| 13 |
+
type: reinforcement-learning
|
| 14 |
+
name: reinforcement-learning
|
| 15 |
+
dataset:
|
| 16 |
+
name: FrozenLake-v1-4x4-no_slippery
|
| 17 |
+
type: FrozenLake-v1-4x4-no_slippery
|
| 18 |
+
metrics:
|
| 19 |
+
- type: mean_reward
|
| 20 |
+
value: 1.00 +/- 0.00
|
| 21 |
+
name: mean_reward
|
| 22 |
+
verified: false
|
| 23 |
+
- name: FrozenLake-Q-Learning-Froggy
|
| 24 |
+
results:
|
| 25 |
+
- task:
|
| 26 |
+
type: reinforcement-learning
|
| 27 |
+
name: reinforcement-learning
|
| 28 |
+
dataset:
|
| 29 |
+
name: FrozenLake-v1-4x4-slippery
|
| 30 |
+
type: FrozenLake-v1-4x4-slippery
|
| 31 |
+
metrics:
|
| 32 |
+
- type: mean_reward
|
| 33 |
+
value: 0.74 +/- 0.44
|
| 34 |
+
name: mean_reward
|
| 35 |
+
verified: false
|
| 36 |
---
|
| 37 |
+
|
| 38 |
+
# 🐤🐸 FrozenLake — Q-Learning
|
| 39 |
+
|
| 40 |
+
### Deterministic vs. Stochastic Environments
|
| 41 |
+
|
| 42 |
+
Two tabular Q-Learning agents trained on Gymnasium's **FrozenLake-v1 (4×4)** to explore how environmental uncertainty affects reinforcement learning.
|
| 43 |
+
|
| 44 |
+
This project extends the [Hugging Face Deep RL Course — Unit 2](https://huggingface.co/learn/deep-rl-course/unit2/introduction) with comparative experiments and animated demonstrations.
|
| 45 |
+
|
| 46 |
+
## Meet the Agents
|
| 47 |
+
|
| 48 |
+
| | 🐤 Ducky | 🐸 Froggy |
|
| 49 |
+
|---|---|---|
|
| 50 |
+
| Environment | Non-slippery | Slippery |
|
| 51 |
+
| Transitions | Deterministic | Stochastic |
|
| 52 |
+
| Algorithm | Q-Learning | Q-Learning |
|
| 53 |
+
| Training Episodes | 10,000 | 10,000 |
|
| 54 |
+
| Evaluation Success Rate | **100%** | **73.7%** |
|
| 55 |
+
|
| 56 |
+
Both agents use identical hyperparameters, so the only difference between them is the environment's transition dynamics.
|
| 57 |
+
|
| 58 |
+
## 🐤🐸 Watch the Agents Play
|
| 59 |
+
|
| 60 |
+
<table>
|
| 61 |
+
<tr>
|
| 62 |
+
<th align="center">🐤 Ducky — Non-Slippery</th>
|
| 63 |
+
<th align="center">🐸 Froggy — Slippery</th>
|
| 64 |
+
</tr>
|
| 65 |
+
<tr>
|
| 66 |
+
<td align="center">
|
| 67 |
+
<img src="./assets/ducky.gif" width="300" alt="Ducky gameplay">
|
| 68 |
+
</td>
|
| 69 |
+
<td align="center">
|
| 70 |
+
<img src="./assets/froggy.gif" width="300" alt="Froggy gameplay">
|
| 71 |
+
</td>
|
| 72 |
+
</tr>
|
| 73 |
+
</table>
|
| 74 |
+
|
| 75 |
+
Ducky follows the same predictable path every time, while Froggy has to deal with slippery ice that can push it in unintended directions. Both animations show successful episodes using the agents' greedy policies.
|
| 76 |
+
|
| 77 |
+
## Training Configuration
|
| 78 |
+
|
| 79 |
+
| Hyperparameter | Value |
|
| 80 |
+
|---|---|
|
| 81 |
+
| Learning Rate (α) | 0.7 |
|
| 82 |
+
| Discount Factor (γ) | 0.95 |
|
| 83 |
+
| Initial Epsilon | 1.0 |
|
| 84 |
+
| Minimum Epsilon | 0.01 |
|
| 85 |
+
| Epsilon Decay | 0.999 (multiplicative, per episode) |
|
| 86 |
+
| Training Episodes | 10,000 |
|
| 87 |
+
| Max Steps per Episode | 100 |
|
| 88 |
+
| Random Seed | 42 |
|
| 89 |
+
|
| 90 |
+
Each agent learns a Q-Table of **16 states × 4 actions** using epsilon-greedy exploration.
|
| 91 |
+
|
| 92 |
+
## Evaluation Results
|
| 93 |
+
|
| 94 |
+
Both agents were evaluated over **1,000 episodes** with greedy action selection and no further Q-Table updates.
|
| 95 |
+
|
| 96 |
+
| Metric | Ducky 🐤 | Froggy 🐸 |
|
| 97 |
+
|---|---:|---:|
|
| 98 |
+
| Success Rate | **100%** | **73.7%** |
|
| 99 |
+
| Mean Reward | 1.000 | 0.737 |
|
| 100 |
+
| Reward Standard Deviation | 0.000 | 0.440 |
|
| 101 |
+
| Average Steps per Episode | 6.00 | 40.57 |
|
| 102 |
+
| Average Steps in Successful Episodes | 6.00 | 36.87 |
|
| 103 |
+
| Success Rate (last 100 training episodes) | 99% | 63% |
|
| 104 |
+
|
| 105 |
+
Training success rates are lower than evaluation success rates because the agents were still exploring (ε > 0) during training.
|
| 106 |
+
|
| 107 |
+
**Ducky** reaches the goal in the optimal 6 steps every time. **Froggy** cannot guarantee success even with a well-learned policy: on the slippery map, the intended move happens only one-third of the time. Its 73.7% success rate is close to the best achievable on this map within a 100-step limit, so the gap reflects the environment, not a failure to learn.
|
| 108 |
+
|
| 109 |
+
These results measure performance on the same map used during training, not generalization to unseen maps.
|
| 110 |
+
|
| 111 |
+
## Key Takeaway
|
| 112 |
+
|
| 113 |
+
**The same Q-Learning algorithm can produce substantially different policies and performance depending on the environment's transition dynamics.**
|
| 114 |
+
|
| 115 |
+
In a deterministic world, the shortest path is the best path. In a stochastic one, the agent learns longer but safer routes that reduce the risk of slipping into a hole.
|
| 116 |
+
|
| 117 |
+
## Usage
|
| 118 |
+
|
| 119 |
+
The repository contains two trained Q-Tables:
|
| 120 |
+
|
| 121 |
+
```text
|
| 122 |
+
models/
|
| 123 |
+
├── ducky_q_table.npy
|
| 124 |
+
└── froggy_q_table.npy
|
| 125 |
+
```
|
| 126 |
+
|
| 127 |
+
Both are compatible with Gymnasium's default 4×4 FrozenLake-v1 state and action spaces.
|
| 128 |
+
|
| 129 |
+
```python
|
| 130 |
+
import numpy as np
|
| 131 |
+
import gymnasium as gym
|
| 132 |
+
from huggingface_hub import hf_hub_download
|
| 133 |
+
|
| 134 |
+
repo_id = "zeynpakn/FrozenLake-Q-Learning"
|
| 135 |
+
|
| 136 |
+
q_table = np.load(hf_hub_download(repo_id=repo_id, filename="models/froggy_q_table.npy"))
|
| 137 |
+
env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True) # use is_slippery=False for Ducky
|
| 138 |
+
|
| 139 |
+
state, _ = env.reset(seed=42)
|
| 140 |
+
done = False
|
| 141 |
+
while not done:
|
| 142 |
+
action = int(np.argmax(q_table[state])) # greedy action
|
| 143 |
+
state, reward, terminated, truncated, _ = env.step(action)
|
| 144 |
+
done = terminated or truncated
|
| 145 |
+
|
| 146 |
+
print("Reached the goal!" if reward == 1 else "Fell into a hole or ran out of steps.")
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
---
|
| 150 |
+
|
| 151 |
+
<p align="center">
|
| 152 |
+
<i>🐤 Ducky follows the rules. Froggy negotiates with physics. 🐸</i>
|
| 153 |
+
</p>
|
| 154 |
+
|
| 155 |
+
---
|
| 156 |
+
|
| 157 |
+
*Built as an educational reinforcement learning experiment, inspired by the Hugging Face Deep RL Course.*
|