zeynpakn commited on
Commit
4e46fc9
·
verified ·
1 Parent(s): 037efe8

Update README.md

Browse files

Center gameplay GIFs in model README

Files changed (1) hide show
  1. README.md +154 -0
README.md CHANGED
@@ -1,3 +1,157 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ tags:
4
+ - reinforcement-learning
5
+ - q-learning
6
+ - frozenlake
7
+ - gymnasium
8
+ - tabular-q-learning
9
+ model-index:
10
+ - name: FrozenLake-Q-Learning-Ducky
11
+ results:
12
+ - task:
13
+ type: reinforcement-learning
14
+ name: reinforcement-learning
15
+ dataset:
16
+ name: FrozenLake-v1-4x4-no_slippery
17
+ type: FrozenLake-v1-4x4-no_slippery
18
+ metrics:
19
+ - type: mean_reward
20
+ value: 1.00 +/- 0.00
21
+ name: mean_reward
22
+ verified: false
23
+ - name: FrozenLake-Q-Learning-Froggy
24
+ results:
25
+ - task:
26
+ type: reinforcement-learning
27
+ name: reinforcement-learning
28
+ dataset:
29
+ name: FrozenLake-v1-4x4-slippery
30
+ type: FrozenLake-v1-4x4-slippery
31
+ metrics:
32
+ - type: mean_reward
33
+ value: 0.74 +/- 0.44
34
+ name: mean_reward
35
+ verified: false
36
  ---
37
+
38
+ # 🐤🐸 FrozenLake — Q-Learning
39
+
40
+ ### Deterministic vs. Stochastic Environments
41
+
42
+ Two tabular Q-Learning agents trained on Gymnasium's **FrozenLake-v1 (4×4)** to explore how environmental uncertainty affects reinforcement learning.
43
+
44
+ This project extends the [Hugging Face Deep RL Course — Unit 2](https://huggingface.co/learn/deep-rl-course/unit2/introduction) with comparative experiments and animated demonstrations.
45
+
46
+ ## Meet the Agents
47
+
48
+ | | 🐤 Ducky | 🐸 Froggy |
49
+ |---|---|---|
50
+ | Environment | Non-slippery | Slippery |
51
+ | Transitions | Deterministic | Stochastic |
52
+ | Algorithm | Q-Learning | Q-Learning |
53
+ | Training Episodes | 10,000 | 10,000 |
54
+ | Evaluation Success Rate | **100%** | **73.7%** |
55
+
56
+ Both agents use identical hyperparameters, so the only difference between them is the environment's transition dynamics.
57
+
58
+ ## 🐤🐸 Watch the Agents Play
59
+
60
+ <table>
61
+ <tr>
62
+ <th align="center">🐤 Ducky — Non-Slippery</th>
63
+ <th align="center">🐸 Froggy — Slippery</th>
64
+ </tr>
65
+ <tr>
66
+ <td align="center">
67
+ <img src="./assets/ducky.gif" width="300" alt="Ducky gameplay">
68
+ </td>
69
+ <td align="center">
70
+ <img src="./assets/froggy.gif" width="300" alt="Froggy gameplay">
71
+ </td>
72
+ </tr>
73
+ </table>
74
+
75
+ Ducky follows the same predictable path every time, while Froggy has to deal with slippery ice that can push it in unintended directions. Both animations show successful episodes using the agents' greedy policies.
76
+
77
+ ## Training Configuration
78
+
79
+ | Hyperparameter | Value |
80
+ |---|---|
81
+ | Learning Rate (α) | 0.7 |
82
+ | Discount Factor (γ) | 0.95 |
83
+ | Initial Epsilon | 1.0 |
84
+ | Minimum Epsilon | 0.01 |
85
+ | Epsilon Decay | 0.999 (multiplicative, per episode) |
86
+ | Training Episodes | 10,000 |
87
+ | Max Steps per Episode | 100 |
88
+ | Random Seed | 42 |
89
+
90
+ Each agent learns a Q-Table of **16 states × 4 actions** using epsilon-greedy exploration.
91
+
92
+ ## Evaluation Results
93
+
94
+ Both agents were evaluated over **1,000 episodes** with greedy action selection and no further Q-Table updates.
95
+
96
+ | Metric | Ducky 🐤 | Froggy 🐸 |
97
+ |---|---:|---:|
98
+ | Success Rate | **100%** | **73.7%** |
99
+ | Mean Reward | 1.000 | 0.737 |
100
+ | Reward Standard Deviation | 0.000 | 0.440 |
101
+ | Average Steps per Episode | 6.00 | 40.57 |
102
+ | Average Steps in Successful Episodes | 6.00 | 36.87 |
103
+ | Success Rate (last 100 training episodes) | 99% | 63% |
104
+
105
+ Training success rates are lower than evaluation success rates because the agents were still exploring (ε > 0) during training.
106
+
107
+ **Ducky** reaches the goal in the optimal 6 steps every time. **Froggy** cannot guarantee success even with a well-learned policy: on the slippery map, the intended move happens only one-third of the time. Its 73.7% success rate is close to the best achievable on this map within a 100-step limit, so the gap reflects the environment, not a failure to learn.
108
+
109
+ These results measure performance on the same map used during training, not generalization to unseen maps.
110
+
111
+ ## Key Takeaway
112
+
113
+ **The same Q-Learning algorithm can produce substantially different policies and performance depending on the environment's transition dynamics.**
114
+
115
+ In a deterministic world, the shortest path is the best path. In a stochastic one, the agent learns longer but safer routes that reduce the risk of slipping into a hole.
116
+
117
+ ## Usage
118
+
119
+ The repository contains two trained Q-Tables:
120
+
121
+ ```text
122
+ models/
123
+ ├── ducky_q_table.npy
124
+ └── froggy_q_table.npy
125
+ ```
126
+
127
+ Both are compatible with Gymnasium's default 4×4 FrozenLake-v1 state and action spaces.
128
+
129
+ ```python
130
+ import numpy as np
131
+ import gymnasium as gym
132
+ from huggingface_hub import hf_hub_download
133
+
134
+ repo_id = "zeynpakn/FrozenLake-Q-Learning"
135
+
136
+ q_table = np.load(hf_hub_download(repo_id=repo_id, filename="models/froggy_q_table.npy"))
137
+ env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True) # use is_slippery=False for Ducky
138
+
139
+ state, _ = env.reset(seed=42)
140
+ done = False
141
+ while not done:
142
+ action = int(np.argmax(q_table[state])) # greedy action
143
+ state, reward, terminated, truncated, _ = env.step(action)
144
+ done = terminated or truncated
145
+
146
+ print("Reached the goal!" if reward == 1 else "Fell into a hole or ran out of steps.")
147
+ ```
148
+
149
+ ---
150
+
151
+ <p align="center">
152
+ <i>🐤 Ducky follows the rules. Froggy negotiates with physics. 🐸</i>
153
+ </p>
154
+
155
+ ---
156
+
157
+ *Built as an educational reinforcement learning experiment, inspired by the Hugging Face Deep RL Course.*