value-based 实践：Q-Learning 智能体（FrozenLake + Taxi） + 回放视频 + 评估分数

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+logs/frozenlake4x4/train_curve.png filter=lfs diff=lfs merge=lfs -text
+logs/frozenlake4x4_slippery/train_curve.png filter=lfs diff=lfs merge=lfs -text
+logs/frozenlake8x8/train_curve.png filter=lfs diff=lfs merge=lfs -text
+logs/taxi/train_curve.png filter=lfs diff=lfs merge=lfs -text
+videos/replay_frozenlake4x4.mp4 filter=lfs diff=lfs merge=lfs -text
+videos/replay_frozenlake4x4_slippery.mp4 filter=lfs diff=lfs merge=lfs -text
+videos/replay_frozenlake8x8.mp4 filter=lfs diff=lfs merge=lfs -text
+videos/replay_taxi.mp4 filter=lfs diff=lfs merge=lfs -text
+videos/replay_value_based_all.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,96 @@
+# Value-based 实践 · Q-Learning（FrozenLake-v1 + Taxi-v3）
+
+> 《第3周-深度强化学习原理实践》PPT 第 27–40 页 value-based 实践产物：**从零手写 Q-Learning**（Q 表 + epsilon-greedy + Bellman 更新），训练好的智能体 + 训练代码 + 回放视频 + 评估分数。
+
+## 1. 评估分数
+
+| 智能体 | 环境 | 平均回报（100 局） | 成功率 | 平均步数 | 训练回合 |
+|---|---|---|---|---|---|
+| FrozenLake-v1 · 4x4 不滑 | `FrozenLake-v1` | **1.00 ± 0.00** | 100% | 6.0 | 10000 |
+| FrozenLake-v1 · 4x4 滑冰版 | `FrozenLake-v1` | **0.74 ± 0.44** | 74% | 39.4 | 20000 |
+| FrozenLake-v1 · 8x8 | `FrozenLake-v1` | **1.00 ± 0.00** | 100% | 14.0 | 60000 |
+| Taxi-v3 | `Taxi-v3` | **7.56 ± 2.71** | 100% | 13.4 | 25000 |
+
+> 评估方式：独立环境、贪心策略 `argmax_a Q(s,a)`、每局最多 99 步，跑 100 局；FrozenLake 的回报是「是否到达目标」（0/1），Taxi 的回报是「每步 −1，成功送达 +20」。
+
+## 2. 回放视频
+
+**总回放（三个智能体合集）**
+
+<video controls width="100%" src="videos/replay_value_based_all.mp4"></video>
+
+**FrozenLake-v1 · 4x4 不滑**
+
+<video controls width="100%" src="videos/replay_frozenlake4x4.mp4"></video>
+
+**FrozenLake-v1 · 4x4 滑冰版**
+
+<video controls width="100%" src="videos/replay_frozenlake4x4_slippery.mp4"></video>
+
+**FrozenLake-v1 · 8x8**
+
+<video controls width="100%" src="videos/replay_frozenlake8x8.mp4"></video>
+
+**Taxi-v3**
+
+<video controls width="100%" src="videos/replay_taxi.mp4"></video>
+
+## 3. 学到的策略（Q 表可视化）
+
+## 4. 训练曲线
+
+![FrozenLake-v1 · 4x4 不滑 训练曲线](logs/frozenlake4x4/train_curve.png)
+
+![FrozenLake-v1 · 4x4 滑冰版 训练曲线](logs/frozenlake4x4_slippery/train_curve.png)
+
+![FrozenLake-v1 · 8x8 训练曲线](logs/frozenlake8x8/train_curve.png)
+
+![Taxi-v3 训练曲线](logs/taxi/train_curve.png)
+
+## 5. 怎么复现
+
+```powershell
+# 环境（本机已有）：conda activate DRL
+conda activate DRL
+
+# 训练（每个实验 1~30 秒）
+python train_qlearning.py --env frozenlake4x4
+python train_qlearning.py --env frozenlake4x4_slippery
+python train_qlearning.py --env frozenlake8x8
+python train_qlearning.py --env taxi
+
+# 评估 + 录视频（生成 eval_results/*.json 和 videos/*.mp4）
+python evaluate_and_record.py --model models/qtable_frozenlake4x4.pkl
+python evaluate_and_record.py --model models/qtable_frozenlake4x4_slippery.pkl --video-episodes 5 --slowmo 4
+python evaluate_and_record.py --model models/qtable_frozenlake8x8.pkl
+python evaluate_and_record.py --model models/qtable_taxi.pkl
+
+# Q 表可视化 + 总回放 + 打包
+python visualize_qtable.py --all
+python make_combined_video.py
+python build_hub_package.py
+```
+
+## 6. 文件说明
+
+- `models/qtable_*.pkl`：模型文件。pickle 格式，内含 Q 表、超参数、训练种子和评估指标；
+  `evaluate_and_record.py` 可以直接加载（不需要重新训练）。
+- `videos/*.mp4`：回放，每帧标注步数/动作/奖励/Q 值条形图，片头片尾是标题卡和分数卡。
+- `eval_results/*.json`：评估分数 + 逐局明细（起点种子、回报、步数、是否成功）。
+- `logs/<run>/train_console.log`：完整训练日志（环境自检、超参数、每 N 回合进度、每次评估）。
+- `code/`：全部代码（训练 / 评估录像 / 可视化 / 打包 / 上传 / 一键脚本）。
+
+## 7. 对应 PPT 页码
+
+| PPT 页 | 内容 | 对应实现 |
+|---|---|---|
+| 29 | Q 表 / Q 函数 / 最优策略 | `train_qlearning.py` 的 `initialize_q_table` / `greedy_policy` |
+| 31–32 | FrozenLake 观测/动作/奖励 | 训练日志里的「环境自检」 |
+| 33 | Q 表初始化 + 贪心策略 | `initialize_q_table` / `greedy_policy` |
+| 34 | epsilon-greedy + 超参数 | `epsilon_greedy_policy` + `EXPERIMENTS` 配置 |
+| 35 | 训练循环（Bellman 更新） | `train_qlearning.py::train` 里的 `Q[s][a] += lr*(...)` |
+| 36–37 | 训练、Q 表可视化、评估、保存模型 | 训练日志 / `visualize_qtable.py` / `evaluate_and_record.py` / `models/*.pkl` |
+| 38–40 | Taxi-v3（500 状态 × 6 动作） | `--env taxi`（含 PPT 的 100 个官方评估种子） |
+| 28 | 上传 Hub + 视频 + 分数 | `build_hub_package.py` + `upload_to_hub.py` + README |
+
+- 生成时间：2026-10-30 23:05

code/build_hub_package.py

@@ -0,0 +1,286 @@
+# -*- coding: utf-8 -*-
+"""
+把训练好的智能体 + 代码 + 视频 + 评估分数打包成"可以直接上传 Hub"的文件夹。
+================================================================
+对应 PPT 第 28 页学习目标第 4 条：
+  "能够将训练好的智能体和代码上传到 Hub，并附上一个漂亮的视频回放和评估分数"
+
+产物 hub_package/ 结构：
+  README.md                    自动填好分数 + 内嵌视频的模型卡
+  metrics.json                 三/四个智能体的评估分数汇总
+  requirements.txt
+  models/qtable_*.pkl          训练出的模型文件（Q 表 + 超参 + 指标）
+  videos/replay_*.mp4          回放视频（含总回放 replay_value_based_all.mp4）
+  eval_results/eval_*.json     评估分数（逐局明细）
+  logs/<run>/train_console.log 训练日志
+  logs/<run>/train_curve.png   训练曲线
+  assets/qtable_policy_*.png   学到的策略可视化
+  code/                        训练 + 评估 + 可视化 + 上传脚本（可复现）
+  run_all.ps1                  一键复现脚本
+
+用法：
+  D:\\study\\robot\\pyenv\\DRL\\python.exe build_hub_package.py
+  D:\\study\\robot\\pyenv\\DRL\\python.exe build_hub_package.py --repo-id 你的用户名/value-based-qlearning
+"""
+
+import argparse
+import json
+import os
+import shutil
+from datetime import datetime
+
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+
+# 要打包的智能体（按 PPT 顺序）
+AGENTS = [
+    dict(run="frozenlake4x4", title="FrozenLake-v1 · 4x4 不滑",
+         desc="PPT 第一部分主实验：确定性冰面，从零手写 Q 表，10,000 回合"),
+    dict(run="frozenlake4x4_slippery", title="FrozenLake-v1 · 4x4 滑冰版",
+         desc="对照实验：只改 is_slippery=True，冰面会打滑，20,000 回合"),
+    dict(run="frozenlake8x8", title="FrozenLake-v1 · 8x8",
+         desc="进阶实验：地图 8x8 + 拉长探索期，60,000 回合"),
+    dict(run="taxi", title="Taxi-v3",
+         desc="PPT 第二部分主实验：500 状态 × 6 动作，25,000 回合，官方 100 个评估种子"),
+]
+
+CODE_FILES = [
+    "train_qlearning.py",
+    "evaluate_and_record.py",
+    "visualize_qtable.py",
+    "make_combined_video.py",
+    "build_hub_package.py",
+    "upload_to_hub.py",
+    "run_all.ps1",
+    "01_实践3说明.md",
+]
+
+REQUIREMENTS = """gymnasium>=1.0
+numpy>=1.24
+imageio>=2.31
+imageio-ffmpeg>=0.4
+Pillow>=9.0
+matplotlib>=3.7
+tqdm>=4.65
+"""
+
+
+def read_json(path):
+    if not os.path.exists(path):
+        return None
+    with open(path, encoding="utf-8") as f:
+        return json.load(f)
+
+
+def build_readme(repo_id=None, scores=None, now=None):
+    raw = f"https://huggingface.co/{repo_id}/resolve/main/" if repo_id else ""
+    link = (lambda p: raw + p) if repo_id else (lambda p: p)
+
+    lines = []
+    lines.append("# Value-based 实践 · Q-Learning（FrozenLake-v1 + Taxi-v3）")
+    lines.append("")
+    lines.append("> 《第3周-深度强化学习原理实践》PPT 第 27–40 页 value-based 实践产物："
+                 "**从零手写 Q-Learning**（Q 表 + epsilon-greedy + Bellman 更新），"
+                 "训练好的智能体 + 训练代码 + 回放视频 + 评估分数。")
+    lines.append("")
+    lines.append("## 1. 评估分数")
+    lines.append("")
+    lines.append("| 智能体 | 环境 | 平均回报（100 局） | 成功率 | 平均步数 | 训练回合 |")
+    lines.append("|---|---|---|---|---|---|")
+    for agent in AGENTS:
+        s = (scores or {}).get(agent["run"])
+        if s:
+            lines.append(
+                f"| {agent['title']} | `{s['env_id']}` | **{s['mean_reward']:.2f} ± "
+                f"{s['std_reward']:.2f}** | {s['success_rate'] * 100:.0f}% | "
+                f"{s['mean_steps']:.1f} | {s.get('n_training_episodes', '—')} |"
+            )
+        else:
+            lines.append(f"| {agent['title']} | — | 待运行 | — | — | — |")
+    lines.append("")
+    lines.append("> 评估方式：独立环境、贪心策略 `argmax_a Q(s,a)`、每局最多 99 步，跑 100 局；"
+                 "FrozenLake 的回报是「是否到达目标」（0/1），Taxi 的回报是"
+                 "「每步 −1，成功送达 +20」。")
+    lines.append("")
+
+    lines.append("## 2. 回放视频")
+    lines.append("")
+    if os.path.exists(os.path.join(BASE_DIR, "videos", "replay_value_based_all.mp4")):
+        lines.append("**总回放（三个智能体合集）**")
+        lines.append("")
+        lines.append(f'<video controls width="100%" src="{link("videos/replay_value_based_all.mp4")}"></video>')
+        lines.append("")
+    for agent in AGENTS:
+        video = f"videos/replay_{agent['run']}.mp4"
+        if os.path.exists(os.path.join(BASE_DIR, video)):
+            lines.append(f"**{agent['title']}**")
+            lines.append("")
+            lines.append(f'<video controls width="100%" src="{link(video)}"></video>')
+            lines.append("")
+
+    lines.append("## 3. 学到的策略���Q 表可视化）")
+    lines.append("")
+    for agent in AGENTS:
+        fig = f"assets/qtable_policy_{agent['run']}.png"
+        if os.path.exists(os.path.join(BASE_DIR, fig)):
+            lines.append(f"![{agent['title']}]({link(fig)})")
+            lines.append("")
+
+    lines.append("## 4. 训练曲线")
+    lines.append("")
+    for agent in AGENTS:
+        fig = f"logs/{agent['run']}/train_curve.png"
+        if os.path.exists(os.path.join(BASE_DIR, fig)):
+            lines.append(f"![{agent['title']} 训练曲线]({link(fig)})")
+            lines.append("")
+
+    lines.append("## 5. 怎么复现")
+    lines.append("")
+    lines.append("```powershell")
+    lines.append("# 环境（本机已有）：conda activate DRL")
+    lines.append("conda activate DRL")
+    lines.append("")
+    lines.append("# 训练（每个实验 1~30 秒）")
+    lines.append("python train_qlearning.py --env frozenlake4x4")
+    lines.append("python train_qlearning.py --env frozenlake4x4_slippery")
+    lines.append("python train_qlearning.py --env frozenlake8x8")
+    lines.append("python train_qlearning.py --env taxi")
+    lines.append("")
+    lines.append("# 评估 + 录视频（生成 eval_results/*.json 和 videos/*.mp4）")
+    lines.append("python evaluate_and_record.py --model models/qtable_frozenlake4x4.pkl")
+    lines.append("python evaluate_and_record.py --model models/qtable_frozenlake4x4_slippery.pkl --video-episodes 5 --slowmo 4")
+    lines.append("python evaluate_and_record.py --model models/qtable_frozenlake8x8.pkl")
+    lines.append("python evaluate_and_record.py --model models/qtable_taxi.pkl")
+    lines.append("")
+    lines.append("# Q 表可视化 + 总回放 + 打包")
+    lines.append("python visualize_qtable.py --all")
+    lines.append("python make_combined_video.py")
+    lines.append("python build_hub_package.py")
+    lines.append("```")
+    lines.append("")
+    lines.append("## 6. 文件说明")
+    lines.append("")
+    lines.append("- `models/qtable_*.pkl`：模型文件。pickle 格式，内含 Q 表、超参数、训练种子和评估指标；")
+    lines.append("  `evaluate_and_record.py` 可以直接加载（不需要重新训练）。")
+    lines.append("- `videos/*.mp4`：回放，每帧标注步数/动作/奖励/Q 值条形图，片头片尾是标题卡和分数卡。")
+    lines.append("- `eval_results/*.json`：评估分数 + 逐局明细（起点种子、回报、步数、是否成功）。")
+    lines.append("- `logs/<run>/train_console.log`：完整训练日志（环境自检、超参数、每 N 回合进度、每次评估）。")
+    lines.append("- `code/`：全部代码（训练 / 评估录像 / 可视化 / 打包 / 上传 / 一键脚本）。")
+    lines.append("")
+    lines.append("## 7. 对应 PPT 页码")
+    lines.append("")
+    lines.append("| PPT 页 | 内容 | 对应实现 |")
+    lines.append("|---|---|---|")
+    lines.append("| 29 | Q 表 / Q 函数 / 最优策略 | `train_qlearning.py` 的 `initialize_q_table` / `greedy_policy` |")
+    lines.append("| 31–32 | FrozenLake 观测/动作/奖励 | 训练日志里的「环境自检」 |")
+    lines.append("| 33 | Q 表初始化 + 贪心策略 | `initialize_q_table` / `greedy_policy` |")
+    lines.append("| 34 | epsilon-greedy + 超参数 | `epsilon_greedy_policy` + `EXPERIMENTS` 配置 |")
+    lines.append("| 35 | 训练循环（Bellman 更新） | `train_qlearning.py::train` 里的 `Q[s][a] += lr*(...)` |")
+    lines.append("| 36–37 | 训练、Q 表可视化、评估、保存模型 | 训练日志 / `visualize_qtable.py` / `evaluate_and_record.py` / `models/*.pkl` |")
+    lines.append("| 38–40 | Taxi-v3（500 状态 × 6 动作） | `--env taxi`（含 PPT 的 100 个官方评估种子） |")
+    lines.append("| 28 | 上传 Hub + 视频 + 分数 | `build_hub_package.py` + `upload_to_hub.py` + README |")
+    lines.append("")
+    lines.append(f"- 生成时间：{now:%Y-%m-%d %H:%M}")
+    lines.append("")
+    return "\n".join(lines)
+
+
+def copy_file(src, dst):
+    os.makedirs(os.path.dirname(dst), exist_ok=True)
+    shutil.copy2(src, dst)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="打包成可上传 Hub 的文件夹")
+    parser.add_argument("--repo-id", type=str, default=None,
+                        help="你的 HF 用户名/仓库名，例如 zhangsan/value-based-qlearning")
+    parser.add_argument("--out", type=str, default=os.path.join(BASE_DIR, "hub_package"))
+    args = parser.parse_args()
+
+    out = args.out
+    if os.path.exists(out):
+        shutil.rmtree(out)
+    os.makedirs(out)
+    now = datetime.now()
+
+    # 1) 读取所有评估分数
+    scores = {}
+    for agent in AGENTS:
+        run = agent["run"]
+        s = read_json(os.path.join(BASE_DIR, "eval_results", f"eval_{run}.json"))
+        if s is None:
+            print(f"[警告] 缺少 eval_results/eval_{run}.json，"
+                  f"请先跑 evaluate_and_record.py --model models/qtable_{run}.pkl")
+            continue
+        scores[run] = s
+    metrics = {}
+    for run, s in scores.items():
+        metrics[run] = {k: s[k] for k in
+                        ("env_id", "n_episodes", "mean_reward", "std_reward",
+                         "success_rate", "mean_steps", "qtable_shape",
+                         "qtable_nonzero", "evaluated_at")}
+
+    # 2) 拷模型
+    for agent in AGENTS:
+        src = os.path.join(BASE_DIR, "models", f"qtable_{agent['run']}.pkl")
+        if os.path.exists(src):
+            copy_file(src, os.path.join(out, "models", os.path.basename(src)))
+        else:
+            print(f"[警告] 缺少模型 {src}")
+
+    # 3) 拷视频（含总回放）
+    video_dir = os.path.join(BASE_DIR, "videos")
+    if os.path.isdir(video_dir):
+        for name in sorted(os.listdir(video_dir)):
+            if name.endswith(".mp4"):
+                copy_file(os.path.join(video_dir, name),
+                          os.path.join(out, "videos", name))
+
+    # 4) 拷评估 json
+    for agent in AGENTS:
+        src = os.path.join(BASE_DIR, "eval_results", f"eval_{agent['run']}.json")
+        if os.path.exists(src):
+            copy_file(src, os.path.join(out, "eval_results", os.path.basename(src)))
+
+    # 5) 拷日志 + 训练曲线
+    for agent in AGENTS:
+        run_dir = os.path.join(BASE_DIR, "logs", agent["run"])
+        for name in ("train_console.log", "train_curve.png", "progress.csv"):
+            src = os.path.join(run_dir, name)
+            if os.path.exists(src):
+                copy_file(src, os.path.join(out, "logs", agent["run"], name))
+
+    # 6) 拷 Q 表可视化
+    for agent in AGENTS:
+        src = os.path.join(BASE_DIR, "assets", f"qtable_policy_{agent['run']}.png")
+        if os.path.exists(src):
+            copy_file(src, os.path.join(out, "assets", os.path.basename(src)))
+
+    # 7) 拷代码
+    for name in CODE_FILES:
+        src = os.path.join(BASE_DIR, name)
+        if os.path.exists(src):
+            copy_file(src, os.path.join(out, "code", name))
+
+    # 8) 写 README / metrics / requirements
+    with open(os.path.join(out, "README.md"), "w", encoding="utf-8") as f:
+        f.write(build_readme(repo_id=args.repo_id, scores=scores, now=now))
+    with open(os.path.join(out, "metrics.json"), "w", encoding="utf-8") as f:
+        json.dump(metrics, f, ensure_ascii=False, indent=2)
+    with open(os.path.join(out, "requirements.txt"), "w", encoding="utf-8") as f:
+        f.write(REQUIREMENTS)
+
+    n_files = sum(len(files) for _, _, files in os.walk(out))
+    print(f"[完成] hub_package/ 共 {n_files} 个文件")
+    for root, dirs, files in os.walk(out):
+        rel = os.path.relpath(root, out)
+        for name in sorted(files):
+            path = os.path.join(root, name)
+            print(f"  {os.path.join(rel, name) if rel != '.' else name}"
+                  f"  ({os.path.getsize(path) / 1024:.0f} KB)")
+    print("\n下一步上传：")
+    print("  hf auth login                       # 粘贴你的 write token")
+    print("  python upload_to_hub.py --repo-id 你的用户名/value-based-qlearning")
+
+
+if __name__ == "__main__":
+    main()

code/evaluate_and_record.py

@@ -0,0 +1,542 @@
+# -*- coding: utf-8 -*-
+"""
+第3周 · 实战3（value-based / Q-Learning）—— 评估 + 录视频脚本
+=============================================================
+对应 PPT 第 36–37 页（评估方法、Q 表可视化、模型文件）与第 28 页学习目标第 4 条：
+  "能够将训练好的智能体和代码上传到 Hub，并附上一个漂亮的视频回放和评估分数"
+
+做三件事：
+  1) 加载训练好的模型文件（models/qtable_*.pkl，里面同时存着 Q 表和全套配置）
+  2) 用贪婪策略独立评估 N 局，输出评估分数（平均回报 ± 标准差、成功率、平均步数）
+     -> eval_results/eval_<run>.json
+  3) 回放录像：每一帧都标注"第几局/第几步/动作/奖励/Q 值条形图/累计回报"，
+     片头有标题卡，片尾有分数卡
+     -> videos/replay_<run>.mp4
+
+用法：
+  D:\\study\\robot\\pyenv\\DRL\\python.exe evaluate_and_record.py --model models/qtable_frozenlake4x4.pkl
+  D:\\study\\robot\\pyenv\\DRL\\python.exe evaluate_and_record.py --model models/qtable_taxi.pkl --episodes 100
+  # 想录得更慢/更多局：
+  python evaluate_and_record.py --model models/qtable_taxi.pkl --video-episodes 10 --slowmo 8
+"""
+
+import argparse
+import json
+import os
+import pickle
+import time
+from datetime import datetime
+
+import gymnasium as gym
+import imageio
+import numpy as np
+from PIL import Image, ImageDraw, ImageFont
+
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+EVAL_DIR = os.path.join(BASE_DIR, "eval_results")
+VIDEO_DIR = os.path.join(BASE_DIR, "videos")
+
+# ----------------------------------------------------------------------
+# 录像画面参数（1280x720 / 30fps，B 站、Hub、PPT 里都合适）
+# ----------------------------------------------------------------------
+WIDTH, HEIGHT = 1280, 720
+HEADER_H = 78
+FOOTER_H = 52
+FPS = 30
+SLOWMO = 6                 # 每个动作帧重复次数（让 Q-Learning 这种短回合看得清）
+HOLD_LAST = 1.2            # 每局结束后的定格秒数
+TITLE_SECONDS = 2.5
+END_SECONDS = 4.0
+
+# 配色
+BG = (15, 23, 42)
+PANEL = (30, 41, 59)
+PANEL_EDGE = (51, 65, 85)
+TEXT = (226, 232, 240)
+MUTED = (148, 163, 184)
+ACCENT = (56, 189, 248)
+GREEN = (34, 197, 94)
+RED = (239, 68, 68)
+AMBER = (245, 158, 11)
+
+FONT_DIR = r"C:\Windows\Fonts"
+
+
+def load_font(size, bold=False):
+    """优先微软雅黑，找不到再退回黑体/默认字体。"""
+    names = ["msyhbd.ttc", "msyh.ttc"] if bold else ["msyh.ttc", "simhei.ttf"]
+    for name in names:
+        path = os.path.join(FONT_DIR, name)
+        if os.path.exists(path):
+            try:
+                return ImageFont.truetype(path, size)
+            except Exception:
+                pass
+    return ImageFont.load_default()
+
+
+FONT_TITLE = load_font(40, bold=True)
+FONT_H1 = load_font(28, bold=True)
+FONT_H2 = load_font(22, bold=True)
+FONT_BODY = load_font(20)
+FONT_SMALL = load_font(16)
+FONT_TINY = load_font(14)
+
+# 动作名称（PPT 第 32 页 / 第 38 页）
+ACTION_NAMES = {
+    "FrozenLake-v1": ["← 向左 (0)", "↓ 向下 (1)", "→ 向右 (2)", "↑ 向上 (3)"],
+    "Taxi-v3": ["↓ 向南 (0)", "↑ 向北 (1)", "→ 向东 (2)", "← 向西 (3)",
+                "P 搭载乘客 (4)", "D 送达乘客 (5)"],
+}
+ACTION_SHORT = {
+    "FrozenLake-v1": ["←", "↓", "→", "↑"],
+    "Taxi-v3": ["南", "北", "东", "西", "载", "送"],
+}
+
+# Taxi-v3 地图上的四个上下客点（R 红 / G 绿 / Y 黄 / B 蓝）
+TAXI_LOCS = [(0, 0), (0, 4), (4, 0), (4, 3)]
+TAXI_LOC_NAMES = ["R 红", "G 绿", "Y 黄", "B 蓝"]
+
+
+# ======================================================================
+# 模型 / 环境
+# ======================================================================
+def load_model(path):
+    with open(path, "rb") as f:
+        model = pickle.load(f)
+    if "qtable" not in model:
+        raise ValueError(f"{path} 里没有 qtable 字段，不是本脚本保存的模型")
+    return model
+
+
+def make_env(model, render_mode="rgb_array"):
+    kwargs = dict(model.get("env_kwargs") or {})
+    kwargs.pop("render_mode", None)
+    return gym.make(model["env_id"], render_mode=render_mode, **kwargs)
+
+
+def greedy_action(q_table, state):
+    return int(np.argmax(q_table[state]))
+
+
+# ======================================================================
+# 评估：贪婪策略跑 N 局（PPT 第 36 页 evaluate_agent 的同一件事）
+# ======================================================================
+def evaluate(env, model, n_episodes, seeds):
+    q = model["qtable"]
+    max_steps = model["max_steps"]
+    rows = []
+    for i in range(n_episodes):
+        if seeds is not None and i < len(seeds):
+            state, _ = env.reset(seed=int(seeds[i]))
+        else:
+            state, _ = env.reset()
+        total, steps = 0.0, 0
+        for step in range(max_steps):
+            action = greedy_action(q, state)
+            state, reward, terminated, truncated, _ = env.step(action)
+            total += float(reward)
+            steps = step + 1
+            if terminated or truncated:
+                break
+        rows.append(dict(episode=i + 1, seed=None if seeds is None else int(seeds[i]),
+                         reward=round(total, 4), steps=steps, success=int(total > 0)))
+    rewards = np.array([r["reward"] for r in rows], dtype=float)
+    success = np.array([r["success"] for r in rows], dtype=float)
+    steps = np.array([r["steps"] for r in rows], dtype=float)
+    result = dict(
+        run=model.get("run_name", "unknown"),
+        env_id=model["env_id"],
+        env_kwargs=model.get("env_kwargs", {}),
+        n_training_episodes=model.get("n_training_episodes"),
+        n_episodes=n_episodes,
+        mean_reward=float(rewards.mean()),
+        std_reward=float(rewards.std()),
+        min_reward=float(rewards.min()),
+        max_reward=float(rewards.max()),
+        success_rate=float(success.mean()),
+        mean_steps=float(steps.mean()),
+        policy="greedy (argmax Q)",
+        model_file=os.path.relpath(model["_path"], BASE_DIR) if "_path" in model else None,
+        qtable_shape=list(q.shape),
+        qtable_nonzero=int(np.count_nonzero(q)),
+        evaluated_at=datetime.now().isoformat(timespec="seconds"),
+        episodes=rows,
+    )
+    return result
+
+
+# ======================================================================
+# 画面工具
+# ======================================================================
+def rounded_rect(draw, box, radius=16, fill=PANEL, outline=PANEL_EDGE, width=2):
+    draw.rounded_rectangle(box, radius=radius, fill=fill, outline=outline, width=width)
+
+
+def draw_text(draw, xy, text, font=FONT_BODY, fill=TEXT, anchor="la"):
+    draw.text(xy, text, font=font, fill=fill, anchor=anchor)
+
+
+def fit_image(frame, max_w, max_h, nearest=True):
+    """把 env.render() 的一帧缩放后居中贴到画布上（像素风用最近邻更清晰）。"""
+    img = Image.fromarray(np.asarray(frame, dtype=np.uint8))
+    scale = min(max_w / img.width, max_h / img.height)
+    new_size = (max(1, int(img.width * scale)), max(1, int(img.height * scale)))
+    resample = Image.NEAREST if nearest else Image.LANCZOS
+    return img.resize(new_size, resample)
+
+
+def base_canvas():
+    canvas = Image.new("RGB", (WIDTH, HEIGHT), BG)
+    draw = ImageDraw.Draw(canvas)
+    # 顶栏
+    draw.rectangle([0, 0, WIDTH, HEADER_H], fill=(15, 23, 42))
+    draw.line([0, HEADER_H, WIDTH, HEADER_H], fill=PANEL_EDGE, width=2)
+    # 底栏
+    draw.rectangle([0, HEIGHT - FOOTER_H, WIDTH, HEIGHT], fill=(15, 23, 42))
+    draw.line([0, HEIGHT - FOOTER_H, WIDTH, HEIGHT - FOOTER_H], fill=PANEL_EDGE, width=2)
+    return canvas, draw
+
+
+def title_card(model, result):
+    """片头标题卡。"""
+    canvas, draw = base_canvas()
+    draw_text(draw, (WIDTH // 2, 150), "Q-Learning 智能体 · 训练后回放", FONT_TITLE,
+              fill=ACCENT, anchor="mm")
+    draw_text(draw, (WIDTH // 2, 215), model.get("title", model["env_id"]), FONT_H1,
+              fill=TEXT, anchor="mm")
+    draw.line([WIDTH // 2 - 220, 260, WIDTH // 2 + 220, 260], fill=PANEL_EDGE, width=2)
+
+    # 中间：三个信息块
+    blocks = [
+        ("算法", "Q-Learning（从零手写 Q 表）"),
+        ("训练规模", f"{model['n_training_episodes']:,} 回合"),
+        ("Q 表", f"{model['qtable'].shape[0]} 状态 × {model['qtable'].shape[1]} 动作"),
+        ("超参数", f"lr={model['learning_rate']}  γ={model['gamma']}  "
+                   f"ε {model['max_epsilon']}→{model['min_epsilon']}"),
+    ]
+    y = 310
+    for k, v in blocks:
+        draw_text(draw, (WIDTH // 2 - 300, y), f"{k}", FONT_SMALL, fill=MUTED)
+        draw_text(draw, (WIDTH // 2 - 160, y), f"{v}", FONT_BODY, fill=TEXT)
+        y += 46
+
+    draw_text(draw, (WIDTH // 2, 570),
+              f"独立评估 {result['n_episodes']} 局："
+              f"平均回报 {result['mean_reward']:.2f} ± {result['std_reward']:.2f}   "
+              f"成功率 {result['success_rate'] * 100:.0f}%",
+              FONT_H2, fill=GREEN, anchor="mm")
+    draw_text(draw, (WIDTH // 2, 615),
+              "对应《第3周-深度强化学习原理实践》PPT 第 27–40 页 · value-based 实践",
+              FONT_SMALL, fill=MUTED, anchor="mm")
+    draw_text(draw, (WIDTH // 2, 640), "模型文件：models/" + os.path.basename(model["_path"]),
+              FONT_SMALL, fill=MUTED, anchor="mm")
+    return canvas
+
+
+def score_card(model, result):
+    """片尾分数卡。"""
+    canvas, draw = base_canvas()
+    draw_text(draw, (WIDTH // 2, 120), "评估结果", FONT_TITLE, fill=ACCENT, anchor="mm")
+    draw_text(draw, (WIDTH // 2, 185), model.get("title", model["env_id"]), FONT_H1,
+              fill=TEXT, anchor="mm")
+
+    # 大号分数
+    mean, std = result["mean_reward"], result["std_reward"]
+    draw_text(draw, (WIDTH // 2 - 200, 300), "平均回报", FONT_BODY, fill=MUTED, anchor="mm")
+    draw_text(draw, (WIDTH // 2 - 200, 360), f"{mean:.2f}", load_font(64, bold=True),
+              fill=GREEN if mean > 0 else RED, anchor="mm")
+    draw_text(draw, (WIDTH // 2 - 200, 410), f"± {std:.2f}", FONT_H2, fill=MUTED, anchor="mm")
+
+    draw_text(draw, (WIDTH // 2 + 60, 300), "成功率", FONT_BODY, fill=MUTED, anchor="mm")
+    draw_text(draw, (WIDTH // 2 + 60, 360), f"{result['success_rate'] * 100:.0f}%",
+              load_font(64, bold=True), fill=ACCENT, anchor="mm")
+
+    draw_text(draw, (WIDTH // 2 + 300, 300), "平均步数", FONT_BODY, fill=MUTED, anchor="mm")
+    draw_text(draw, (WIDTH // 2 + 300, 360), f"{result['mean_steps']:.1f}",
+              load_font(64, bold=True), fill=AMBER, anchor="mm")
+
+    draw_text(draw, (WIDTH // 2, 505),
+              f"{result['n_episodes']} 局独立评估 · 贪婪策略 argmax_a Q(s,a)",
+              FONT_BODY, fill=TEXT, anchor="mm")
+    # 迷你分数分布
+    rewards = [e["reward"] for e in result["episodes"]]
+    values, counts = np.unique(np.round(rewards, 2), return_counts=True)
+    draw_text(draw, (WIDTH // 2, 555),
+              "逐局回报分布：" + "  ".join(f"{v:g}×{c}" for v, c in zip(values[:8], counts[:8]))
+              + (" …" if len(values) > 8 else ""),
+              FONT_SMALL, fill=MUTED, anchor="mm")
+    draw_text(draw, (WIDTH // 2, 620),
+              "模型与代码已打包上传 Hub：附视频回放 + 评估分数",
+              FONT_SMALL, fill=MUTED, anchor="mm")
+    return canvas
+
+
+def draw_q_bars(draw, q_row, action_names, chosen_action, x, y, width):
+    """把当前状态各动作的 Q 值画成水平条形图（直观看出贪心策略选了什么）。"""
+    q = np.asarray(q_row, dtype=float)
+    lo, hi = float(q.min()), float(q.max())
+    span = hi - lo if hi - lo > 1e-9 else 1.0
+    bar_h = 26
+    gap = 10
+    for a in range(len(q)):
+        yy = y + a * (bar_h + gap)
+        chosen = a == chosen_action
+        fill = ACCENT if chosen else (71, 85, 105)
+        # 动作标签
+        draw_text(draw, (x, yy + bar_h // 2), action_names[a], FONT_SMALL,
+                  fill=TEXT if chosen else MUTED, anchor="lm")
+        # 条形
+        left = x + 150
+        draw.rectangle([left, yy, left + width, yy + bar_h], fill=(51, 65, 85))
+        frac = (q[a] - lo) / span
+        draw.rectangle([left, yy, left + max(3, int(width * frac)), yy + bar_h], fill=fill)
+        # 数值
+        draw_text(draw, (left + width + 12, yy + bar_h // 2), f"{q[a]:.3f}", FONT_SMALL,
+                  fill=ACCENT if chosen else MUTED, anchor="lm")
+        if chosen:
+            draw_text(draw, (left + width + 78, yy + bar_h // 2), "← 最优", FONT_SMALL,
+                      fill=ACCENT, anchor="lm")
+
+
+def game_frame(model, env, state, step, episode_idx, n_video_eps, total_reward,
+               last_reward, action, ended, success, eval_result):
+    """合成一帧：左边是环境画面，右边是 Q 值/动作/奖励信息。"""
+    canvas, draw = base_canvas()
+
+    title = model.get("title", model["env_id"])
+    draw_text(draw, (32, HEADER_H // 2), f"Q-Learning 智能体回放 · {title}",
+              FONT_H2, fill=TEXT, anchor="lm")
+    draw_text(draw, (WIDTH - 32, HEADER_H // 2),
+              "PPT 第 27–40 页 · value-based", FONT_SMALL, fill=MUTED, anchor="rm")
+
+    # ---------- 左边：环境画面 ----------
+    box = [32, HEADER_H + 22, 672, HEIGHT - FOOTER_H - 22]
+    rounded_rect(draw, box)
+    raw = env.render()
+    if raw is not None:
+        inner_w = box[2] - box[0] - 24
+        inner_h = box[3] - box[1] - 24
+        img = fit_image(raw, inner_w, inner_h)
+        px = box[0] + (box[2] - box[0] - img.width) // 2
+        py = box[1] + (box[3] - box[1] - img.height) // 2
+        canvas.paste(img, (px, py))
+
+    # ---------- 右边：信息面板 ----------
+    px0, px1 = 700, WIDTH - 32
+    rounded_rect(draw, [px0, HEADER_H + 22, px1, HEIGHT - FOOTER_H - 22])
+
+    x = px0 + 24
+    y = HEADER_H + 50
+    draw_text(draw, (x, y), f"第 {episode_idx + 1} / {n_video_eps} 局", FONT_H2, fill=TEXT)
+    y += 34
+
+    # Taxi 额外显示乘客/目的地
+    if model["env_id"] == "Taxi-v3":
+        taxi_row, taxi_col, pass_idx, dest_idx = env.unwrapped.decode(int(state))
+        pass_text = "在车上" if pass_idx == 4 else TAXI_LOC_NAMES[pass_idx] + " 处"
+        draw_text(draw, (x, y), f"出租车  ({taxi_row},{taxi_col})  乘客 {pass_text}  "
+                                f"目的地 {TAXI_LOC_NAMES[dest_idx]}", FONT_SMALL, fill=MUTED)
+        y += 30
+    else:
+        draw_text(draw, (x, y), f"状态编号 s = {int(state)}", FONT_SMALL, fill=MUTED)
+        y += 30
+
+    # 步数 / 动作 / 奖励
+    draw_text(draw, (x, y), "步数", FONT_SMALL, fill=MUTED)
+    draw_text(draw, (x + 70, y), f"{step}", FONT_H2, fill=TEXT)
+    draw_text(draw, (x + 180, y), "动作", FONT_SMALL, fill=MUTED)
+    draw_text(draw, (x + 250, y),
+              ACTION_NAMES[model["env_id"]][action] if action is not None else "—",
+              FONT_H2, fill=ACCENT)
+    y += 38
+
+    draw_text(draw, (x, y), "单步奖励", FONT_SMALL, fill=MUTED)
+    draw_text(draw, (x + 120, y), f"{last_reward:+.2f}" if last_reward is not None else "—",
+              FONT_H2, fill=GREEN if (last_reward or 0) > 0 else TEXT)
+    draw_text(draw, (x + 260, y), "累计回报", FONT_SMALL, fill=MUTED)
+    draw_text(draw, (x + 360, y), f"{total_reward:+.2f}", FONT_H2,
+              fill=GREEN if total_reward > 0 else TEXT)
+    y += 44
+
+    # Q 值条形图（终局状态不再更新 Q，改成提示文字）
+    if ended:
+        draw_text(draw, (x, y), "终局状态：回合结束，Q 表在这一帧不再更新", FONT_SMALL, fill=MUTED)
+    else:
+        draw_text(draw, (x, y), f"Q(s={int(state)}, ·)  ← 贪心策略从这里选动作", FONT_SMALL, fill=MUTED)
+        y += 30
+        q = model["qtable"][int(state)]
+        chosen = int(np.argmax(q))
+        bars_width = 180
+        draw_q_bars(draw, q, ACTION_SHORT[model["env_id"]], chosen, x, y, bars_width)
+
+    # 回合结束横幅
+    if ended:
+        banner_y = HEIGHT - FOOTER_H - 105
+        if model["env_id"] == "Taxi-v3":
+            text, color = ("√ 成功送达乘客（+20 奖励）", GREEN) if success else \
+                          ("× 超时 / 未送达（回合结束）", RED)
+        else:
+            text, color = ("√ 成功到达目标（+1 奖励）", GREEN) if success else \
+                          ("× 掉进冰洞（奖励 0，回合结束）", RED)
+        draw.rounded_rectangle([px0 + 20, banner_y, px1 - 20, banner_y + 62],
+                               radius=12, fill=(color[0] // 4, color[1] // 4, color[2] // 4),
+                               outline=color, width=2)
+        draw_text(draw, ((px0 + px1) // 2, banner_y + 31), text, FONT_H2, fill=color, anchor="mm")
+
+    # ---------- 底栏：总体评估分数 ----------
+    draw_text(draw, (32, HEIGHT - FOOTER_H // 2),
+              f"评估 {eval_result['n_episodes']} 局：平均回报 "
+              f"{eval_result['mean_reward']:.2f} ± {eval_result['std_reward']:.2f}   "
+              f"成功率 {eval_result['success_rate'] * 100:.0f}%   "
+              f"平均步数 {eval_result['mean_steps']:.1f}",
+              FONT_SMALL, fill=MUTED, anchor="lm")
+    draw_text(draw, (WIDTH - 32, HEIGHT - FOOTER_H // 2), "贪心策略 argmax_a Q(s,a)",
+              FONT_SMALL, fill=MUTED, anchor="rm")
+    return canvas
+
+
+# ======================================================================
+# 录像
+# ======================================================================
+def record_video(env, model, result, out_path, n_video_episodes, slowmo, fps):
+    q = model["qtable"]
+    max_steps = model["max_steps"]
+    seeds = model.get("eval_seed")
+    writer = imageio.get_writer(out_path, fps=fps, codec="libx264", quality=8,
+                                macro_block_size=2)
+    n_frames = 0
+
+    def push(img, repeat=1):
+        nonlocal n_frames
+        frame = np.asarray(img, dtype=np.uint8)
+        for _ in range(repeat):
+            writer.append_data(frame)
+            n_frames += 1
+
+    # 片头
+    push(title_card(model, result), int(TITLE_SECONDS * fps))
+    hold_end = int(HOLD_LAST * fps)
+
+    for ep in range(n_video_episodes):
+        if seeds is not None and ep < len(seeds):
+            state, _ = env.reset(seed=int(seeds[ep]))
+        else:
+            state, _ = env.reset()
+
+        total_reward = 0.0
+        last_reward = None
+        prev_reward = None
+        last_action = None
+        step = 0
+        for step in range(max_steps):
+            action = greedy_action(q, state)
+            last_action = action
+            new_state, reward, terminated, truncated, _ = env.step(action)
+            done = terminated or truncated
+            total_reward += float(reward)
+            success = total_reward > 0
+            last_reward = float(reward)
+            # 这一步的"决策画面"：画面里的状态是"还没执行这个动作之前"的状态，
+            # 所以累计回报用执行前的值，单步奖励显示上一步拿到的奖励
+            push(game_frame(model, env, state, step, ep, n_video_episodes,
+                            total_reward - last_reward, prev_reward, action, False, success, result),
+                 slowmo)
+            prev_reward = last_reward
+            state = new_state
+            if done:
+                break
+        # 回合结束：定格 + 结束横幅
+        ended_frame = game_frame(model, env, state, step + 1, ep, n_video_episodes,
+                                 total_reward, last_reward, last_action, True,
+                                 total_reward > 0, result)
+        push(ended_frame, hold_end)
+        print(f"  [录像] 第 {ep + 1}/{n_video_episodes} 局：回报={total_reward:+.2f}，"
+              f"步数={step + 1}，{'√ 成功' if total_reward > 0 else '× 失败'}")
+
+    # 片尾分数卡
+    push(score_card(model, result), int(END_SECONDS * fps))
+    writer.close()
+    return n_frames
+
+
+# ======================================================================
+# 主流程
+# ======================================================================
+def main():
+    parser = argparse.ArgumentParser(description="评估 Q-Learning 智能体并录制回放视频")
+    parser.add_argument("--model", type=str, required=True, help="models/qtable_*.pkl")
+    parser.add_argument("--episodes", type=int, default=None,
+                        help="评估局数（默认用模型里记录的 n_eval_episodes）")
+    parser.add_argument("--video-episodes", type=int, default=8, help="录几局回放（默认 8）")
+    parser.add_argument("--slowmo", type=int, default=SLOWMO, help="每个动作帧重复几次（默认 6）")
+    parser.add_argument("--fps", type=int, default=FPS, help="视频帧率（默认 30）")
+    parser.add_argument("--out-video", type=str, default=None, help="视频输出路径（.mp4）")
+    parser.add_argument("--out-json", type=str, default=None, help="评估结果输出路径（.json）")
+    parser.add_argument("--skip-video", action="store_true",
+                        help="只评估、不录视频（改完分数想快速重算时用）")
+    args = parser.parse_args()
+
+    model = load_model(args.model)
+    model["_path"] = os.path.abspath(args.model)
+    model["run_name"] = os.path.splitext(os.path.basename(args.model))[0].replace("qtable_", "")
+    if not model.get("title"):
+        model["title"] = model["env_id"]
+
+    os.makedirs(EVAL_DIR, exist_ok=True)
+    os.makedirs(VIDEO_DIR, exist_ok=True)
+
+    run = model["run_name"]
+    out_json = args.out_json or os.path.join(EVAL_DIR, f"eval_{run}.json")
+    out_video = args.out_video or os.path.join(VIDEO_DIR, f"replay_{run}.mp4")
+
+    n_eval = args.episodes or model.get("n_eval_episodes", 100)
+    seeds = model.get("eval_seed")
+    if seeds is not None and len(seeds) < n_eval:
+        seeds = None   # 种子不够就不指定，退回随机起点
+
+    print("=" * 78)
+    print(f"评估 + 录像 · {model['title']}")
+    print(f"模型文件：{os.path.relpath(model['_path'], BASE_DIR)}")
+    print(f"贪婪策略：argmax_a Q(s,a)   Q 表形状：{model['qtable'].shape}")
+    print("=" * 78)
+
+    env = make_env(model)
+    print(f"\n[评估] 用贪婪策略跑 {n_eval} 局（独立环境，起点种子："
+          f"{'模型自带的 ' + str(len(seeds)) + ' 个种子' if seeds is not None else '随机'}）")
+    started = time.time()
+    result = evaluate(env, model, n_eval, seeds)
+    result["evaluation_seconds"] = round(time.time() - started, 2)
+    print(f"  Mean_reward = {result['mean_reward']:.2f} +/- {result['std_reward']:.2f}")
+    print(f"  成功率      = {result['success_rate'] * 100:.0f}%")
+    print(f"  平均步数    = {result['mean_steps']:.1f}")
+    print(f"  回报范围    = [{result['min_reward']:.2f}, {result['max_reward']:.2f}]")
+    with open(out_json, "w", encoding="utf-8") as f:
+        json.dump(result, f, ensure_ascii=False, indent=2)
+    print(f"[输出] 评估分数 = {os.path.relpath(out_json, BASE_DIR)}")
+
+    if args.skip_video:
+        print("\n[跳过录像]（--skip-video）")
+        env.close()
+        print("\n" + "=" * 78)
+        print(f"完成。评估分数 {result['mean_reward']:.2f} ± {result['std_reward']:.2f}，"
+              f"成功率 {result['success_rate'] * 100:.0f}%")
+        print("=" * 78)
+        return
+
+    print(f"\n[录像] 录 {args.video_episodes} 局回放，每帧慢放 ×{args.slowmo}，{args.fps}fps")
+    n_frames = record_video(env, model, result, out_video, args.video_episodes,
+                            args.slowmo, args.fps)
+    secs = n_frames / args.fps
+    print(f"[输出] 回放视频 = {os.path.relpath(out_video, BASE_DIR)} "
+          f"({n_frames} 帧 / {secs:.1f} 秒 / {os.path.getsize(out_video) / 1024:.0f} KB)")
+    env.close()
+
+    print("\n" + "=" * 78)
+    print(f"完成。评估分数 {result['mean_reward']:.2f} ± {result['std_reward']:.2f}，"
+          f"成功率 {result['success_rate'] * 100:.0f}%")
+    print("=" * 78)
+
+
+if __name__ == "__main__":
+    main()

code/make_combined_video.py

@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+"""
+把三个智能体的回放视频拼成一支"总回放"视频（方便上传 Hub / 交作业）。
+==============================================================
+输入：videos/replay_frozenlake4x4.mp4、replay_frozenlake8x8.mp4、replay_taxi.mp4
+输出：videos/replay_value_based_all.mp4（1280x720 / 30fps，开头有总览卡，结尾有分数汇总卡）
+
+用法：
+  D:\\study\\robot\\pyenv\\DRL\\python.exe make_combined_video.py
+"""
+
+import json
+import os
+
+import imageio.v2 as imageio
+import numpy as np
+
+import evaluate_and_record as ev   # 复用录像脚本里的画布/字体/配色
+
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+VIDEO_DIR = os.path.join(BASE_DIR, "videos")
+EVAL_DIR = os.path.join(BASE_DIR, "eval_results")
+
+FPS = ev.FPS
+RUNS = [
+    ("frozenlake4x4", "FrozenLake-v1 · 4x4 不滑", "PPT 第一部分（10,000 回合）"),
+    ("frozenlake8x8", "FrozenLake-v1 · 8x8", "进阶：地图变大 + 探索调度"),
+    ("taxi", "Taxi-v3", "PPT 第二部分（25,000 回合）"),
+]
+
+
+def load_scores():
+    scores = {}
+    for run, _, _ in RUNS:
+        path = os.path.join(EVAL_DIR, f"eval_{run}.json")
+        if os.path.exists(path):
+            with open(path, encoding="utf-8") as f:
+                scores[run] = json.load(f)
+    return scores
+
+
+def intro_card(scores):
+    canvas, draw = ev.base_canvas()
+    ev.draw_text(draw, (ev.WIDTH // 2, 110), "value-based 实践 · Q-Learning 智能体回放合集",
+                 ev.FONT_TITLE, fill=ev.ACCENT, anchor="mm")
+    ev.draw_text(draw, (ev.WIDTH // 2, 165),
+                 "从零手写 Q 表 + epsilon-greedy + Bellman 更新（PPT 第 27–40 页）",
+                 ev.FONT_BODY, fill=ev.TEXT, anchor="mm")
+    draw.line([120, 205, ev.WIDTH - 120, 205], fill=ev.PANEL_EDGE, width=2)
+
+    y = 265
+    for i, (run, name, note) in enumerate(RUNS):
+        s = scores.get(run)
+        ev.draw_text(draw, (110, y), f"{i + 1}. {name}", ev.FONT_H2, fill=ev.TEXT)
+        ev.draw_text(draw, (110, y + 34), note, ev.FONT_SMALL, fill=ev.MUTED)
+        if s:
+            score_text = f"{s['mean_reward']:.2f} ± {s['std_reward']:.2f}"
+            ev.draw_text(draw, (ev.WIDTH - 110, y + 16), score_text, ev.FONT_H1,
+                         fill=ev.GREEN if s["mean_reward"] > 0 else ev.RED, anchor="rm")
+            ev.draw_text(draw, (ev.WIDTH - 110, y + 46),
+                         f"成功率 {s['success_rate'] * 100:.0f}%   "
+                         f"平均步数 {s['mean_steps']:.1f}",
+                         ev.FONT_SMALL, fill=ev.MUTED, anchor="rm")
+        y += 95
+
+    ev.draw_text(draw, (ev.WIDTH // 2, 620),
+                 "每段回放都标注了步数 / 动作 / Q 值条形图，片尾有该智能体的评估分数",
+                 ev.FONT_SMALL, fill=ev.MUTED, anchor="mm")
+    return canvas
+
+
+def end_card(scores):
+    canvas, draw = ev.base_canvas()
+    ev.draw_text(draw, (ev.WIDTH // 2, 105), "三个智能体评估分数汇总", ev.FONT_TITLE,
+                 fill=ev.ACCENT, anchor="mm")
+    ev.draw_text(draw, (ev.WIDTH // 2, 155),
+                 "独立环境 · 贪婪策略 argmax_a Q(s,a) · 100 局", ev.FONT_BODY,
+                 fill=ev.MUTED, anchor="mm")
+    draw.line([120, 195, ev.WIDTH - 120, 195], fill=ev.PANEL_EDGE, width=2)
+
+    y = 265
+    for run, name, _ in RUNS:
+        s = scores.get(run)
+        if not s:
+            continue
+        ev.draw_text(draw, (120, y), name, ev.FONT_H2, fill=ev.TEXT)
+        ev.draw_text(draw, (560, y), f"回报 {s['mean_reward']:.2f} ± {s['std_reward']:.2f}",
+                     ev.FONT_H2, fill=ev.GREEN if s["mean_reward"] > 0 else ev.RED)
+        ev.draw_text(draw, (880, y), f"成功率 {s['success_rate'] * 100:.0f}%",
+                     ev.FONT_H2, fill=ev.ACCENT)
+        y += 90
+
+    ev.draw_text(draw, (ev.WIDTH // 2, 610),
+                 "模型文件（Q 表 pkl）+ 训练代码 + 日志 已打包上传 Hub",
+                 ev.FONT_BODY, fill=ev.TEXT, anchor="mm")
+    return canvas
+
+
+def main():
+    scores = load_scores()
+    clips = []
+    for run, _, _ in RUNS:
+        path = os.path.join(VIDEO_DIR, f"replay_{run}.mp4")
+        if not os.path.exists(path):
+            print(f"[跳过] 找不到 {path}，请先跑 evaluate_and_record.py --model models/qtable_{run}.pkl")
+            continue
+        clips.append(path)
+    if not clips:
+        raise SystemExit("没有可拼接的视频。")
+
+    out_path = os.path.join(VIDEO_DIR, "replay_value_based_all.mp4")
+    writer = imageio.get_writer(out_path, fps=FPS, codec="libx264", quality=8,
+                                macro_block_size=2)
+    frames = 0
+
+    def push(img, repeat=1):
+        nonlocal frames
+        arr = np.asarray(img, dtype=np.uint8)
+        for _ in range(repeat):
+            writer.append_data(arr)
+            frames += 1
+
+    push(intro_card(scores), int(3.0 * FPS))
+    for path in clips:
+        reader = imageio.get_reader(path)
+        n = 0
+        for frame in reader:
+            writer.append_data(frame)
+            frames += 1
+            n += 1
+        reader.close()
+        print(f"  [拼接] {os.path.basename(path)}：{n} 帧")
+    push(end_card(scores), int(5.0 * FPS))
+
+    writer.close()
+    size_mb = os.path.getsize(out_path) / (1024 * 1024)
+    print(f"[输出] 总回放视频 = videos/{os.path.basename(out_path)} "
+          f"({frames} 帧 / {frames / FPS:.1f} 秒 / {size_mb:.1f} MB)")
+
+
+if __name__ == "__main__":
+    main()

code/train_qlearning.py

@@ -0,0 +1,604 @@
+# -*- coding: utf-8 -*-
+"""
+第3周 · 实战3（value-based / Q-Learning）—— 训练脚本
+=====================================================
+对应 PPT《第3周-深度强化学习原理实践》第 27–40 页：
+  · 第 29 页：Q 表 / Q 函数 / 最优策略 pi*(s) = argmax_a Q*(s,a)
+  · 第 31–32 页：FrozenLake-v1（4x4 不滑版）观测/动作/奖励
+  · 第 33 页：Q 表初始化 + 贪心策略
+  · 第 34 页：epsilon-greedy 行为策略 + 超参数
+  · 第 35 页：Q-Learning 训练循环（Bellman 更新）
+  · 第 36–37 页：训练、Q 表可视化、评估
+  · 第 38–40 页：Taxi-v3
+
+本脚本从零手写 Q-Learning（不依赖 stable-baselines3），一次跑通三个环境：
+  frozenlake4x4  FrozenLake-v1 4x4，is_slippery=False（PPT 第一部分，确定性）
+  frozenlake8x8  FrozenLake-v1 8x8，is_slippery=True （PPT 第 31 页提到的进阶版本，随机性）
+  taxi           Taxi-v3（PPT 第二部分，500 个状态 × 6 个动作）
+
+输出（全部落在本目录下）：
+  logs/<run>/train_console.log   完整控制台日志（含环境自检/超参/每 N 回合进度/每次评估）
+  logs/<run>/progress.csv        结构化进度数据（画曲线 / 写报告用）
+  logs/<run>/train_curve.png     训练曲线（回报、epsilon、成功率、评估）
+  models/qtable_<run>.pkl        训练出的模型文件（Q 表 + 全套配置，可被 evaluate_and_record.py 直接加载）
+
+用法（本机已装好 DRL 环境，无需再装包）：
+  D:\\study\\robot\\pyenv\\DRL\\python.exe train_qlearning.py --env frozenlake4x4
+  D:\\study\\robot\\pyenv\\DRL\\python.exe train_qlearning.py --env frozenlake8x8
+  D:\\study\\robot\\pyenv\\DRL\\python.exe train_qlearning.py --env taxi
+
+只改一个变量做对照实验（例如演示"epsilon 衰减太慢"）：
+  python train_qlearning.py --env frozenlake4x4 --decay-rate 0.0001 --run-name fl4x4_slow_decay
+"""
+
+import argparse
+import csv
+import os
+import pickle
+import random
+import sys
+import time
+from datetime import datetime
+
+import gymnasium as gym
+import matplotlib
+matplotlib.use("Agg")               # 不弹窗，直接存 png
+import matplotlib.pyplot as plt
+import numpy as np
+
+# ======================================================================
+# 全局路径
+# ======================================================================
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+LOG_ROOT = os.path.join(BASE_DIR, "logs")
+MODEL_ROOT = os.path.join(BASE_DIR, "models")
+
+# 中文字体：Windows 自带的微软雅黑 / 黑体
+plt.rcParams["font.sans-serif"] = ["Microsoft YaHei", "SimHei", "DejaVu Sans"]
+plt.rcParams["axes.unicode_minus"] = False
+
+# ======================================================================
+# 第 38–39 页：Taxi-v3 官方评估种子（PPT 原文：请勿修改 EVAL_SEED）
+# 作用：让所有同学都在相同的出租车起始位置上评估，班级分数才可比。
+# ======================================================================
+EVAL_SEED_TAXI = [
+    16, 54, 165, 177, 191, 191, 120, 80, 149, 178, 48, 38, 6, 125, 174, 73, 50,
+    172, 100, 148, 146, 6, 25, 40, 68, 148, 49, 167, 9, 97, 164, 176, 61, 7, 54,
+    55, 161, 131, 184, 51, 170, 12, 120, 113, 95, 126, 51, 98, 36, 135, 54, 82,
+    45, 95, 89, 59, 95, 124, 9, 113, 58, 85, 51, 134, 121, 169, 105, 21, 30, 11,
+    50, 65, 12, 43, 82, 145, 152, 97, 106, 55, 31, 85, 38, 112, 102, 168, 123, 97,
+    21, 83, 158, 26, 80, 63, 5, 81, 32, 11, 28, 148,
+]
+
+# ======================================================================
+# 环境 + 超参数配置（严格对应 PPT 第 34 页与第 39 页）
+# ======================================================================
+EXPERIMENTS = {
+    # ---- 第一部分：冰冻湖 4x4，不滑（确定性） -------------------------
+    "frozenlake4x4": dict(
+        title="FrozenLake-v1（4x4，is_slippery=False）",
+        env_id="FrozenLake-v1",
+        env_kwargs=dict(map_name="4x4", is_slippery=False),
+        n_training_episodes=10000,      # 第 34 页：总训练回合数
+        learning_rate=0.7,              # 第 34 页：学习率
+        gamma=0.95,                     # 第 34 页：折扣率
+        max_steps=99,                   # 第 34 页：每回合最大步数
+        max_epsilon=1.0,                # 第 34 页：开始时探索概率
+        min_epsilon=0.05,               # 第 34 页：最小探索概率
+        decay_rate=0.0005,              # 第 34 页：探索概率指数衰减率
+        n_eval_episodes=100,            # 第 34 页：评估回合数
+        eval_seed=None,                 # 第 34 页 eval_seed = []
+        eval_every=1000,
+        log_every=500,
+    ),
+    # ---- 第 31 页提到的另一种模式：4x4 + 滑冰（is_slippery=True） ------
+    # 与第一部分只差一个变量 is_slippery，但冰面会以 1/3 概率把智能体推偏，
+    # 所以同样的超参数明显更难学：这里把探索期拉长（min_epsilon 0.1 / decay 0.0003）
+    # 并多训一倍回合，成功率才能从 ~20% 提到 ~75%。
+    "frozenlake4x4_slippery": dict(
+        title="FrozenLake-v1（4x4，is_slippery=True）",
+        env_id="FrozenLake-v1",
+        env_kwargs=dict(map_name="4x4", is_slippery=True),
+        n_training_episodes=20000,
+        learning_rate=0.7,
+        gamma=0.95,
+        max_steps=99,
+        max_epsilon=1.0,
+        min_epsilon=0.1,
+        decay_rate=0.0003,
+        n_eval_episodes=100,
+        eval_seed=list(range(100)),
+        eval_every=2000,
+        log_every=1000,
+    ),
+    # ---- 第 31 页提到的进阶版：冰冻湖 8x8（地图变大） -----------------
+    # 8x8 最优路径约 14 步，比 4x4 长一倍多；Q 表从 16x4 变成 64x4。
+    # 注意：8x8 的稀疏奖励让"随机探索"很难碰到终点，如果沿用 4x4 的快速 epsilon 衰减，
+    #       Q 表会一直是全 0（这一点在实验报告里作为对照实验记录）。
+    #       所以这里把探索期拉长（min_epsilon 0.3 / decay 0.00003），让智能体先"逛遍"地图。
+    "frozenlake8x8": dict(
+        title="FrozenLake-v1（8x8，is_slippery=False）",
+        env_id="FrozenLake-v1",
+        env_kwargs=dict(map_name="8x8", is_slippery=False),
+        n_training_episodes=60000,
+        learning_rate=0.8,
+        gamma=0.95,
+        max_steps=200,
+        max_epsilon=1.0,
+        min_epsilon=0.3,
+        decay_rate=0.00003,
+        n_eval_episodes=100,
+        eval_seed=list(range(100)),
+        eval_every=6000,
+        log_every=3000,
+    ),
+    # ---- 第二部分：Taxi-v3 -------------------------------------------
+    "taxi": dict(
+        title="Taxi-v3",
+        env_id="Taxi-v3",
+        env_kwargs=dict(),
+        n_training_episodes=25000,      # 第 39 页：总训练回合数
+        learning_rate=0.7,
+        gamma=0.95,
+        max_steps=99,
+        max_epsilon=1.0,
+        min_epsilon=0.05,
+        decay_rate=0.005,               # 第 39 页：0.005（比冰冻湖衰减快，因为回合更长）
+        n_eval_episodes=100,
+        eval_seed=EVAL_SEED_TAXI,       # 第 39 页：官方评估种子
+        eval_every=2500,
+        log_every=1000,
+    ),
+}
+
+
+# ======================================================================
+# 工具：同时写控制台 + 日志文件（这就是"有日志输出的训练代码"）
+# ======================================================================
+class Tee:
+    """把 print 的内容同时写进 stdout 和日志文件。"""
+
+    def __init__(self, log_path):
+        self.terminal = sys.stdout
+        self.log_file = open(log_path, "w", encoding="utf-8")
+
+    def write(self, message):
+        self.terminal.write(message)
+        self.log_file.write(message)
+        self.log_file.flush()
+
+    def flush(self):
+        self.terminal.flush()
+        self.log_file.flush()
+
+
+def timestamp():
+    return datetime.now().strftime("%H:%M:%S")
+
+
+# ======================================================================
+# 第 33 页：创建并初始化 Q 表
+#   Q 表大小 = (状态数, 动作数)，初始全 0（"在开始时 Q 表是无用的"）
+# ======================================================================
+def initialize_q_table(state_space, action_space):
+    return np.zeros((state_space, action_space))
+
+
+# ======================================================================
+# 第 33 页：贪婪策略（更新策略 / 最终策略）
+#   pi*(s) = argmax_a Q(s,a)
+# ======================================================================
+def greedy_policy(q_table, state):
+    return int(np.argmax(q_table[state]))
+
+
+# ======================================================================
+# 第 34 页：epsilon-greedy 策略（行为策略）
+#   以 epsilon 的概率探索（随机动作），否则利用（贪心动作）
+#   —— Q-Learning 是离线策略（off-policy）：行动用 epsilon-greedy，
+#      更新时却用 max_a Q(s', a)，两套策略。
+# ======================================================================
+def epsilon_greedy_policy(q_table, state, epsilon, action_space):
+    if random.random() > epsilon:
+        return greedy_policy(q_table, state)          # 利用
+    return int(action_space.sample())                 # 探索
+
+
+# ======================================================================
+# 第 35 页：训练循环
+#   Q(s,a) <- Q(s,a) + lr * [ r + gamma * max_a' Q(s',a') - Q(s,a) ]
+# ======================================================================
+def train(
+    env,
+    q_table,
+    n_training_episodes,
+    max_steps,
+    learning_rate,
+    gamma,
+    max_epsilon,
+    min_epsilon,
+    decay_rate,
+    train_seed=0,
+    log_every=500,
+    eval_every=None,
+    eval_kwargs=None,
+):
+    """
+    返回：q_table、每回合数据、评估历史
+    """
+    episode_rewards = []
+    episode_steps = []
+    episode_success = []          # 是否到达目标（FrozenLake 的 reward==1 / Taxi 的 reward==20）
+    epsilons = []
+    eval_history = []             # [(episode, mean, std, success_rate), ...]
+
+    started = time.time()
+    eval_kwargs = eval_kwargs or {}
+
+    for episode in range(n_training_episodes):
+        # ---- epsilon 指数衰减：越往后越少探索 ----
+        epsilon = min_epsilon + (max_epsilon - min_epsilon) * np.exp(-decay_rate * episode)
+        epsilons.append(epsilon)
+
+        # ---- 重置环境 ----
+        # 每回合都用固定种子 reset(seed=train_seed+episode)：
+        # 同一个 --seed 重跑，起点序列、滑冰随机性和探索随机性完全一致 -> 分数可复现。
+        # 注意：gymnasium 的 env.reset(seed=...) 只给 env 播种，不给 action_space 播种，
+        #       而 epsilon-greedy 的"探索"要用 env.action_space.sample()，
+        #       所以这里再单独给 action_space 播种一次（这一行是可复现的关键）。
+        if train_seed is not None:
+            state, _ = env.reset(seed=train_seed + episode)
+            env.action_space.seed(train_seed + episode)
+        else:
+            state, _ = env.reset()
+
+        total_reward = 0.0
+        reached_goal = False
+
+        for step in range(max_steps):
+            # 1) 用 epsilon-greedy 选择动作 At
+            action = epsilon_greedy_policy(q_table, state, epsilon, env.action_space)
+
+            # 2) 执行动作，观察 Rt+1 和 St+1
+            new_state, reward, terminated, truncated, _ = env.step(action)
+            done = terminated or truncated
+
+            # 3) Bellman 更新（Q-Learning 的核心这一行）
+            #    注意：真实终止状态没有未来价值，next_q 取 0；
+            #    PPT 截图里没做这个掩码，这里修正后结果更稳（FrozenLake 两者等价）。
+            next_q = 0.0 if terminated else float(np.max(q_table[new_state]))
+            q_table[state][action] += learning_rate * (
+                reward + gamma * next_q - q_table[state][action]
+            )
+
+            total_reward += reward
+            state = new_state
+            if done:
+                reached_goal = reward > 0
+                break
+
+        episode_rewards.append(total_reward)
+        episode_steps.append(step + 1)
+        episode_success.append(1 if reached_goal else 0)
+
+        # ---- 每 log_every 回合打印一行进度（可直接粘进实验报告）----
+        if (episode + 1) % log_every == 0 or episode == 0:
+            window = episode_rewards[-min(100, len(episode_rewards)):]
+            window_success = episode_success[-min(100, len(episode_success)):]
+            window_steps = episode_steps[-min(100, len(episode_steps)):]
+            elapsed_min = (time.time() - started) / 60.0
+            print(
+                f"[train] ep={episode + 1:>6}/{n_training_episodes} | "
+                f"eps={epsilon:.3f} | "
+                f"最近100回合平均回报={np.mean(window):6.2f} | "
+                f"成功率={np.mean(window_success) * 100:5.1f}% | "
+                f"平均步数={np.mean(window_steps):5.1f} | "
+                f"Q非零项={np.count_nonzero(q_table):5d} | "
+                f"用时={elapsed_min:5.2f}min"
+            )
+
+        # ---- 周期评估（用贪婪策略跑独立回合）----
+        if eval_every and (episode + 1) % eval_every == 0:
+            mean_r, std_r, succ = evaluate_agent(
+                env, q_table, max_steps, eval_kwargs.get("n_eval", 20),
+                eval_kwargs.get("eval_seed"),
+            )
+            eval_history.append((episode + 1, mean_r, std_r, succ))
+            print(
+                f"[eval ] ep={episode + 1:>6} | 贪婪策略 {eval_kwargs.get('n_eval', 20)} 局: "
+                f"回报={mean_r:6.2f} ± {std_r:5.2f} | 成功率={succ * 100:5.1f}%"
+            )
+
+    return q_table, episode_rewards, episode_steps, episode_success, epsilons, eval_history
+
+
+# ======================================================================
+# 第 36 页：评估智能体（贪婪策略，返回平均回报 ± 标准差）
+# ======================================================================
+def evaluate_agent(env, q_table, max_steps, n_episodes, seeds=None):
+    rewards = []
+    successes = []
+    lengths = []
+    for i in range(n_episodes):
+        if seeds is not None and i < len(seeds):
+            state, _ = env.reset(seed=int(seeds[i]))
+        else:
+            state, _ = env.reset()
+        total_reward = 0.0
+        for step in range(max_steps):
+            action = greedy_policy(q_table, state)
+            state, reward, terminated, truncated, _ = env.step(action)
+            total_reward += reward
+            if terminated or truncated:
+                break
+        rewards.append(total_reward)
+        lengths.append(step + 1)
+        successes.append(1 if total_reward > 0 else 0)
+    return float(np.mean(rewards)), float(np.std(rewards)), float(np.mean(successes))
+
+
+# ======================================================================
+# 画训练曲线
+# ======================================================================
+def moving_average(values, window=100):
+    values = np.asarray(values, dtype=float)
+    if len(values) < window:
+        return values
+    kernel = np.ones(window) / window
+    return np.convolve(values, kernel, mode="valid")
+
+
+def plot_curve(run_dir, title, episode_rewards, epsilons, episode_success,
+               eval_history, n_training_episodes):
+    fig, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=False)
+
+    # 上图：每回合回报 + 100 回合滑动平均 + 周期评估
+    ax = axes[0]
+    ax.plot(np.arange(1, len(episode_rewards) + 1), episode_rewards,
+            color="#93c5fd", linewidth=0.5, alpha=0.6, label="每回合回报")
+    ma = moving_average(episode_rewards, 100)
+    ax.plot(np.arange(100, 100 + len(ma)), ma,
+            color="#1d4ed8", linewidth=2, label="100 回合滑动平均")
+    if eval_history:
+        xs = [h[0] for h in eval_history]
+        ys = [h[1] for h in eval_history]
+        errs = [h[2] for h in eval_history]
+        ax.errorbar(xs, ys, yerr=errs, fmt="o--", color="#dc2626", markersize=4,
+                    linewidth=1.2, capsize=3, label="周期评估（贪婪策略）")
+    ax.set_xlabel("训练回合 episode")
+    ax.set_ylabel("回报 reward")
+    ax.set_title(f"{title} · 训练曲线（共 {n_training_episodes} 回合）")
+    # 早期有几局极端负分（例如 Taxi 乱按上下客被 -10 罚到 -400），会把曲线压扁，
+    # 这里按 0.5%~99.5% 分位数裁一下 y 轴，极端点仍在图外但趋势看得清。
+    rewards_arr = np.asarray(episode_rewards, dtype=float)
+    low = min(float(np.percentile(rewards_arr, 0.5)), 0.0) - 2
+    high = max(float(np.percentile(rewards_arr, 99.5)), 1.0) + 1
+    if high - low > 5:
+        ax.set_ylim(low, high)
+    ax.legend(loc="lower right", fontsize=9)
+    ax.grid(alpha=0.3)
+
+    # 下图：epsilon 衰减 + 探索/利用的阶段
+    ax2 = axes[1]
+    ax2.plot(np.arange(1, len(epsilons) + 1), epsilons, color="#f59e0b", linewidth=2,
+             label="epsilon（探索概率）")
+    ax2.fill_between(np.arange(1, len(epsilons) + 1), 0, epsilons, color="#f59e0b", alpha=0.15)
+    ax2.set_xlabel("训练回合 episode")
+    ax2.set_ylabel("epsilon")
+    ax2.set_title("epsilon 指数衰减：从\"大胆探索\"到\"相信 Q 表\"")
+    ax2.grid(alpha=0.3)
+    ax2.legend(loc="upper right", fontsize=9)
+
+    fig.tight_layout()
+    path = os.path.join(run_dir, "train_curve.png")
+    fig.savefig(path, dpi=130)
+    plt.close(fig)
+    return path
+
+
+def write_progress_csv(run_dir, rewards, steps, success, epsilons, eval_history):
+    path = os.path.join(run_dir, "progress.csv")
+    with open(path, "w", newline="", encoding="utf-8-sig") as f:
+        writer = csv.writer(f)
+        writer.writerow(["episode", "epsilon", "episode_reward", "episode_steps", "success"])
+        for i, (r, s, ok, eps) in enumerate(zip(rewards, steps, success, epsilons), start=1):
+            writer.writerow([i, f"{eps:.6f}", f"{r:.6f}", s, ok])
+        writer.writerow([])
+        writer.writerow(["eval_episode", "eval_mean_reward", "eval_std_reward", "eval_success_rate"])
+        for ep, mean_r, std_r, succ in eval_history:
+            writer.writerow([ep, f"{mean_r:.6f}", f"{std_r:.6f}", f"{succ:.4f}"])
+    return path
+
+
+# ======================================================================
+# 主流程
+# ======================================================================
+def main():
+    parser = argparse.ArgumentParser(description="Q-Learning 训练（FrozenLake / Taxi）")
+    parser.add_argument("--env", choices=list(EXPERIMENTS.keys()), default="frozenlake4x4",
+                        help="用哪个环境训练")
+    parser.add_argument("--episodes", type=int, default=None, help="覆盖总训练回合数")
+    parser.add_argument("--learning-rate", type=float, default=None, help="覆盖学习率 lr")
+    parser.add_argument("--gamma", type=float, default=None, help="覆盖折扣率 gamma")
+    parser.add_argument("--decay-rate", type=float, default=None, help="覆盖 epsilon 衰减率")
+    parser.add_argument("--min-epsilon", type=float, default=None, help="覆盖最小探索概率")
+    parser.add_argument("--max-steps", type=int, default=None, help="覆盖每回合最大步数")
+    parser.add_argument("--eval-episodes", type=int, default=None, help="覆盖最终评估回合数")
+    parser.add_argument("--eval-every", type=int, default=None, help="每多少回合评估一次")
+    parser.add_argument("--log-every", type=int, default=None, help="每多少回合打印一行日志")
+    parser.add_argument("--seed", type=int, default=0, help="训练随机种子（默认 0）")
+    parser.add_argument("--run-name", type=str, default=None, help="日志/模型文件名前缀")
+    parser.add_argument("--slippery", action="store_true",
+                        help="把 FrozenLake 的冰面切换成滑冰版（is_slippery=True，随机转移）")
+    args = parser.parse_args()
+
+    cfg = dict(EXPERIMENTS[args.env])
+    if args.slippery:
+        cfg["env_kwargs"] = dict(cfg["env_kwargs"], is_slippery=True)
+        cfg["title"] += " · 滑冰版"
+        # 滑冰版起点随机性大，固定 100 个起点种子才能公平比较
+        cfg["eval_seed"] = list(range(100))
+    # 命令行覆盖（方便做"一次只改一个变量"的对照实验）
+    if args.episodes is not None:
+        cfg["n_training_episodes"] = args.episodes
+    if args.learning_rate is not None:
+        cfg["learning_rate"] = args.learning_rate
+    if args.gamma is not None:
+        cfg["gamma"] = args.gamma
+    if args.decay_rate is not None:
+        cfg["decay_rate"] = args.decay_rate
+    if args.min_epsilon is not None:
+        cfg["min_epsilon"] = args.min_epsilon
+    if args.max_steps is not None:
+        cfg["max_steps"] = args.max_steps
+    if args.eval_episodes is not None:
+        cfg["n_eval_episodes"] = args.eval_episodes
+    if args.eval_every is not None:
+        cfg["eval_every"] = args.eval_every
+    if args.log_every is not None:
+        cfg["log_every"] = args.log_every
+    cfg["train_seed"] = args.seed
+
+    run_name = args.run_name or args.env
+    run_dir = os.path.join(LOG_ROOT, run_name)
+    os.makedirs(run_dir, exist_ok=True)
+    os.makedirs(MODEL_ROOT, exist_ok=True)
+
+    # 每次运行先设置随机种子：Python random 管探索、np 管 Q 表初始化
+    random.seed(args.seed)
+    np.random.seed(args.seed)
+
+    tee = Tee(os.path.join(run_dir, "train_console.log"))
+    sys.stdout = tee
+    try:
+        print("=" * 78)
+        print(f"Q-Learning 训练 · {cfg['title']}")
+        print(f"运行名称: {run_name}    开始时间: {datetime.now():%Y-%m-%d %H:%M:%S}")
+        print("=" * 78)
+
+        # ---------------- 第 31–32 页：创建并理解环境 ----------------
+        env = gym.make(cfg["env_id"], render_mode="rgb_array", **cfg["env_kwargs"])
+        state_space = env.observation_space.n
+        action_space = env.action_space.n
+        sample_state, _ = env.reset(seed=args.seed)
+
+        print("\n[环境自检]")
+        print(f"  env_id             = {cfg['env_id']}")
+        print(f"  env_kwargs         = {cfg['env_kwargs']}")
+        print(f"  观测空间           = {env.observation_space}  -> {state_space} 个状态")
+        print(f"  动作空间           = {env.action_space}  -> {action_space} 个动作")
+        if cfg["env_id"] == "FrozenLake-v1":
+            ncol = env.unwrapped.ncol
+            print(f"  一次 reset 的观测  = {sample_state}"
+                  f"（整数 = 状态编号，{env.unwrapped.nrow}x{ncol} 地图按 row*{ncol}+col 编码）")
+        else:
+            print(f"  一次 reset 的观测  = {sample_state}（整数 = 状态编号，共 500 个）")
+        print(f"  每回合最大步数     = {cfg['max_steps']}")
+
+        # ---------------- 第 33 页：Q 表 ----------------
+        q_table = initialize_q_table(state_space, action_space)
+        print("\n[Q 表初始化]")
+        print(f"  Q 表形状 = {q_table.shape}（= (状态数, 动作数)），初始值全为 0")
+        print(f"  需要学习的参数个数 = {state_space * action_space}")
+
+        # ---------------- 第 34 页：超参数 ----------------
+        print("\n[超参数]")
+        print(f"  n_training_episodes = {cfg['n_training_episodes']}")
+        print(f"  learning_rate       = {cfg['learning_rate']}")
+        print(f"  gamma               = {cfg['gamma']}")
+        print(f"  max_steps           = {cfg['max_steps']}")
+        print(f"  max_epsilon         = {cfg['max_epsilon']}")
+        print(f"  min_epsilon         = {cfg['min_epsilon']}")
+        print(f"  decay_rate          = {cfg['decay_rate']}")
+        print(f"  n_eval_episodes     = {cfg['n_eval_episodes']}")
+        print(f"  eval_seed           = {'官方 100 个种子' if cfg['eval_seed'] else '[]（不指定）'}")
+        print(f"  train_seed          = {args.seed}")
+
+        # ---------------- 第 35 页：训练 ----------------
+        print(f"\n[开始训练] {timestamp()}  每 {cfg['log_every']} 回合打印一行，"
+              f"每 {cfg['eval_every']} 回合评估一次")
+        print(f"  可复现性：每回合 env.reset(seed={args.seed}+episode)，同 --seed 重跑分数一致")
+        started = time.time()
+        (q_table, rewards, steps, success, epsilons, eval_history) = train(
+            env=env,
+            q_table=q_table,
+            n_training_episodes=cfg["n_training_episodes"],
+            max_steps=cfg["max_steps"],
+            learning_rate=cfg["learning_rate"],
+            gamma=cfg["gamma"],
+            max_epsilon=cfg["max_epsilon"],
+            min_epsilon=cfg["min_epsilon"],
+            decay_rate=cfg["decay_rate"],
+            train_seed=args.seed,
+            log_every=cfg["log_every"],
+            eval_every=cfg["eval_every"],
+            eval_kwargs=dict(
+                n_eval=20,
+                eval_seed=cfg["eval_seed"],
+            ),
+        )
+        elapsed = (time.time() - started) / 60.0
+        print(f"\n[训练完成] {timestamp()}  总用时 = {elapsed:.2f} 分钟")
+
+        # ---------------- 第 36 页：最终评估 ----------------
+        print(f"\n[最终评估] 用贪婪策略跑 {cfg['n_eval_episodes']} 局（独立于训练）")
+        mean_r, std_r, succ = evaluate_agent(
+            env, q_table, cfg["max_steps"], cfg["n_eval_episodes"], cfg["eval_seed"]
+        )
+        print(f"  Mean_reward = {mean_r:.2f} +/- {std_r:.2f}")
+        print(f"  成功率      = {succ * 100:.0f}%")
+        if eval_history:
+            best = max(eval_history, key=lambda h: h[1])
+            print(f"  训练途中最好的周期评估：{best[1]:.2f} ± {best[2]:.2f} @ 第 {best[0]} 回合")
+        print(f"  Q 表非零项 = {np.count_nonzero(q_table)} / {q_table.size}")
+        print(f"  Q 表最大价值 = {q_table.max():.4f}")
+
+        # ---------------- 第 36–37 页：Q 表可视化 + 训练曲线 ----------------
+        curve_path = plot_curve(run_dir, cfg["title"], rewards, epsilons, success,
+                                eval_history, cfg["n_training_episodes"])
+        csv_path = write_progress_csv(run_dir, rewards, steps, success, epsilons, eval_history)
+        print(f"\n[输出] 训练曲线 = {os.path.relpath(curve_path, BASE_DIR)}")
+        print(f"[输出] 进度 CSV = {os.path.relpath(csv_path, BASE_DIR)}")
+
+        # ---------------- 第 37 页：保存模型（= "训练出的模型文件"） ----------------
+        model = {
+            "run_name": run_name,
+            "title": cfg["title"],
+            "env_id": cfg["env_id"],
+            "env_kwargs": cfg["env_kwargs"],
+            "max_steps": cfg["max_steps"],
+            "n_training_episodes": cfg["n_training_episodes"],
+            "n_eval_episodes": cfg["n_eval_episodes"],
+            "eval_seed": cfg["eval_seed"],
+            "learning_rate": cfg["learning_rate"],
+            "gamma": cfg["gamma"],
+            "max_epsilon": cfg["max_epsilon"],
+            "min_epsilon": cfg["min_epsilon"],
+            "decay_rate": cfg["decay_rate"],
+            "train_seed": args.seed,
+            "qtable": q_table,
+            "metrics": {
+                "mean_reward": mean_r,
+                "std_reward": std_r,
+                "success_rate": succ,
+                "training_minutes": elapsed,
+            },
+            "trained_at": datetime.now().isoformat(timespec="seconds"),
+        }
+        model_path = os.path.join(MODEL_ROOT, f"qtable_{run_name}.pkl")
+        with open(model_path, "wb") as f:
+            pickle.dump(model, f)
+        print(f"[输出] 模型文件 = {os.path.relpath(model_path, BASE_DIR)} "
+              f"({os.path.getsize(model_path) / 1024:.1f} KB)")
+
+        print("\n" + "=" * 78)
+        print(f"完成。下一步：python evaluate_and_record.py --model models/qtable_{run_name}.pkl")
+        print("=" * 78)
+        env.close()
+    finally:
+        sys.stdout = tee.terminal
+        tee.log_file.close()
+
+
+if __name__ == "__main__":
+    main()

code/upload_to_hub.py

@@ -0,0 +1,104 @@
+# -*- coding: utf-8 -*-
+"""
+把 hub_package/ 上传到 Hugging Face Hub（对应 PPT 第 28 页第 4 条学习目标）。
+================================================================
+等价于 PPT 里的 package_to_hub(...)，只是拆成"先打包、再上传"两步，更透明。
+
+准备：
+  1) 注册 https://huggingface.co ，在 Settings -> Access Tokens 建一个 write token
+  2) 登录（二选一）：
+       hf auth login              # 交互式粘贴 token
+       或者直接设环境变量：$env:HF_TOKEN = "hf_xxx"
+  3) 打包：python build_hub_package.py --repo-id 你的用户名/value-based-qlearning
+
+上传：
+  python upload_to_hub.py --repo-id 你的用户名/value-based-qlearning
+  python upload_to_hub.py --repo-id 你的用户名/value-based-qlearning --private
+
+如果本机连不上 huggingface.co（本机实测会超时），有三条路：
+  A) 开代理后重试本脚本；
+  B) 换镜像（只能下载不能上传，不适用于上传）；
+  C) 浏览器手动上传：https://huggingface.co/new 建仓库，把 hub_package/ 里的文件拖进去。
+"""
+
+import argparse
+import os
+import sys
+
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+
+
+def list_files(folder):
+    rows = []
+    for root, _, files in os.walk(folder):
+        for name in sorted(files):
+            path = os.path.join(root, name)
+            rows.append((os.path.relpath(path, folder), os.path.getsize(path)))
+    return rows
+
+
+def main():
+    parser = argparse.ArgumentParser(description="上传 hub_package 到 Hugging Face Hub")
+    parser.add_argument("--repo-id", required=True, help="用户名/仓库名，例如 zhangsan/value-based-qlearning")
+    parser.add_argument("--folder", default=os.path.join(BASE_DIR, "hub_package"))
+    parser.add_argument("--token", default=os.environ.get("HF_TOKEN"), help="默认读环境变量 HF_TOKEN")
+    parser.add_argument("--private", action="store_true", help="创建私有仓库")
+    parser.add_argument("--endpoint", default=os.environ.get("HF_ENDPOINT"),
+                        help="默认 huggingface.co，可用环境变量 HF_ENDPOINT 覆盖")
+    parser.add_argument("--dry-run", action="store_true", help="只列出要上传的文件，不真的上传")
+    args = parser.parse_args()
+
+    if not os.path.isdir(args.folder):
+        raise SystemExit(f"找不到 {args.folder}，请先运行 python build_hub_package.py")
+
+    files = list_files(args.folder)
+    total_mb = sum(size for _, size in files) / (1024 * 1024)
+    print(f"准备上传 {len(files)} 个文件（共 {total_mb:.1f} MB）到仓库 {args.repo_id}")
+    for rel, size in files:
+        print(f"  {rel}  ({size / 1024:.0f} KB)")
+
+    if args.dry_run:
+        print("\n[--dry-run] 不上传。")
+        return
+
+    if not args.token:
+        print("\n[错误] 没有 token。请先执行 hf auth login，或设置 $env:HF_TOKEN。")
+        print("       也可以去掉 --token 参数让 huggingface_hub 读本机缓存的登录信息。")
+        return
+
+    try:
+        from huggingface_hub import HfApi
+    except ImportError:
+        print("\n[错误] 缺 huggingface_hub，请先：pip install huggingface_hub")
+        return
+
+    api = HfApi(endpoint=args.endpoint, token=args.token) if args.endpoint else HfApi(token=args.token)
+
+    try:
+        print(f"\n[1/2] 创建/确认仓库 {args.repo_id} ...")
+        api.create_repo(repo_id=args.repo_id, repo_type="model",
+                        private=args.private, exist_ok=True)
+        print("[2/2] 上传文件 ...")
+        api.upload_folder(
+            repo_id=args.repo_id,
+            repo_type="model",
+            folder_path=args.folder,
+            commit_message="value-based 实践：Q-Learning 智能体（FrozenLake + Taxi）"
+                           " + 回放视频 + 评估分数",
+        )
+    except Exception as exc:                       # noqa: BLE001
+        print(f"\n[上传失败] {type(exc).__name__}: {exc}")
+        print("\n常见原因与对策：")
+        print("  1) 网络连不上 huggingface.co（本机常见）-> 开代理后重试，或浏览器手动上传；")
+        print("  2) token 没有 write 权限 -> 重新建一个 write token；")
+        print("  3) 仓库名格式不对 -> 必须是 用户名/仓库名。")
+        print(f"\n手动上传入口：https://huggingface.co/new  （仓库名建议 value-based-qlearning）")
+        return
+
+    url = f"https://huggingface.co/{args.repo_id}"
+    print(f"\n[完成] 仓库地址：{url}")
+    print(f"       视频和评估分数已经写进 README.md，打开仓库就能看到。")
+
+
+if __name__ == "__main__":
+    sys.exit(main())

eval_results/eval_frozenlake4x4.json

@@ -0,0 +1,727 @@
+{
+  "run": "frozenlake4x4",
+  "env_id": "FrozenLake-v1",
+  "env_kwargs": {
+    "map_name": "4x4",
+    "is_slippery": false
+  },
+  "n_training_episodes": 10000,
+  "n_episodes": 100,
+  "mean_reward": 1.0,
+  "std_reward": 0.0,
+  "min_reward": 1.0,
+  "max_reward": 1.0,
+  "success_rate": 1.0,
+  "mean_steps": 6.0,
+  "policy": "greedy (argmax Q)",
+  "model_file": "models\\qtable_frozenlake4x4.pkl",
+  "qtable_shape": [
+    16,
+    4
+  ],
+  "qtable_nonzero": 35,
+  "evaluated_at": "2026-10-30T22:46:48",
+  "episodes": [
+    {
+      "episode": 1,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 2,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 3,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 4,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 5,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 6,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 7,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 8,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 9,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 10,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 11,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 12,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 13,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 14,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 15,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 16,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 17,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 18,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 19,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 20,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 21,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 22,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 23,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 24,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 25,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 26,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 27,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 28,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 29,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 30,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 31,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 32,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 33,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 34,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 35,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 36,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 37,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 38,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 39,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 40,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 41,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 42,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 43,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 44,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 45,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 46,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 47,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 48,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 49,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 50,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 51,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 52,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 53,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 54,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 55,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 56,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 57,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 58,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 59,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 60,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 61,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 62,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 63,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 64,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 65,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 66,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 67,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 68,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 69,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 70,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 71,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 72,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 73,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 74,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 75,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 76,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 77,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 78,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 79,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 80,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 81,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 82,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 83,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 84,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 85,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 86,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 87,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 88,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 89,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 90,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 91,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 92,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 93,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 94,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 95,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 96,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 97,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 98,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 99,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 100,
+      "seed": null,
+      "reward": 1.0,
+      "steps": 6,
+      "success": 1
+    }
+  ],
+  "evaluation_seconds": 0.01
+}

eval_results/eval_frozenlake4x4_slippery.json

@@ -0,0 +1,727 @@
+{
+  "run": "frozenlake4x4_slippery",
+  "env_id": "FrozenLake-v1",
+  "env_kwargs": {
+    "map_name": "4x4",
+    "is_slippery": true
+  },
+  "n_training_episodes": 20000,
+  "n_episodes": 100,
+  "mean_reward": 0.74,
+  "std_reward": 0.4386342439892262,
+  "min_reward": 0.0,
+  "max_reward": 1.0,
+  "success_rate": 0.74,
+  "mean_steps": 39.41,
+  "policy": "greedy (argmax Q)",
+  "model_file": "models\\qtable_frozenlake4x4_slippery.pkl",
+  "qtable_shape": [
+    16,
+    4
+  ],
+  "qtable_nonzero": 44,
+  "evaluated_at": "2026-10-30T22:46:59",
+  "episodes": [
+    {
+      "episode": 1,
+      "seed": 0,
+      "reward": 0.0,
+      "steps": 54,
+      "success": 0
+    },
+    {
+      "episode": 2,
+      "seed": 1,
+      "reward": 1.0,
+      "steps": 41,
+      "success": 1
+    },
+    {
+      "episode": 3,
+      "seed": 2,
+      "reward": 0.0,
+      "steps": 99,
+      "success": 0
+    },
+    {
+      "episode": 4,
+      "seed": 3,
+      "reward": 1.0,
+      "steps": 27,
+      "success": 1
+    },
+    {
+      "episode": 5,
+      "seed": 4,
+      "reward": 1.0,
+      "steps": 98,
+      "success": 1
+    },
+    {
+      "episode": 6,
+      "seed": 5,
+      "reward": 1.0,
+      "steps": 78,
+      "success": 1
+    },
+    {
+      "episode": 7,
+      "seed": 6,
+      "reward": 1.0,
+      "steps": 24,
+      "success": 1
+    },
+    {
+      "episode": 8,
+      "seed": 7,
+      "reward": 1.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 9,
+      "seed": 8,
+      "reward": 1.0,
+      "steps": 42,
+      "success": 1
+    },
+    {
+      "episode": 10,
+      "seed": 9,
+      "reward": 1.0,
+      "steps": 23,
+      "success": 1
+    },
+    {
+      "episode": 11,
+      "seed": 10,
+      "reward": 1.0,
+      "steps": 96,
+      "success": 1
+    },
+    {
+      "episode": 12,
+      "seed": 11,
+      "reward": 0.0,
+      "steps": 30,
+      "success": 0
+    },
+    {
+      "episode": 13,
+      "seed": 12,
+      "reward": 1.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 14,
+      "seed": 13,
+      "reward": 0.0,
+      "steps": 13,
+      "success": 0
+    },
+    {
+      "episode": 15,
+      "seed": 14,
+      "reward": 1.0,
+      "steps": 18,
+      "success": 1
+    },
+    {
+      "episode": 16,
+      "seed": 15,
+      "reward": 1.0,
+      "steps": 25,
+      "success": 1
+    },
+    {
+      "episode": 17,
+      "seed": 16,
+      "reward": 1.0,
+      "steps": 37,
+      "success": 1
+    },
+    {
+      "episode": 18,
+      "seed": 17,
+      "reward": 1.0,
+      "steps": 24,
+      "success": 1
+    },
+    {
+      "episode": 19,
+      "seed": 18,
+      "reward": 1.0,
+      "steps": 22,
+      "success": 1
+    },
+    {
+      "episode": 20,
+      "seed": 19,
+      "reward": 0.0,
+      "steps": 76,
+      "success": 0
+    },
+    {
+      "episode": 21,
+      "seed": 20,
+      "reward": 1.0,
+      "steps": 34,
+      "success": 1
+    },
+    {
+      "episode": 22,
+      "seed": 21,
+      "reward": 1.0,
+      "steps": 26,
+      "success": 1
+    },
+    {
+      "episode": 23,
+      "seed": 22,
+      "reward": 1.0,
+      "steps": 19,
+      "success": 1
+    },
+    {
+      "episode": 24,
+      "seed": 23,
+      "reward": 1.0,
+      "steps": 23,
+      "success": 1
+    },
+    {
+      "episode": 25,
+      "seed": 24,
+      "reward": 1.0,
+      "steps": 22,
+      "success": 1
+    },
+    {
+      "episode": 26,
+      "seed": 25,
+      "reward": 1.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 27,
+      "seed": 26,
+      "reward": 1.0,
+      "steps": 23,
+      "success": 1
+    },
+    {
+      "episode": 28,
+      "seed": 27,
+      "reward": 0.0,
+      "steps": 30,
+      "success": 0
+    },
+    {
+      "episode": 29,
+      "seed": 28,
+      "reward": 1.0,
+      "steps": 9,
+      "success": 1
+    },
+    {
+      "episode": 30,
+      "seed": 29,
+      "reward": 1.0,
+      "steps": 32,
+      "success": 1
+    },
+    {
+      "episode": 31,
+      "seed": 30,
+      "reward": 0.0,
+      "steps": 31,
+      "success": 0
+    },
+    {
+      "episode": 32,
+      "seed": 31,
+      "reward": 0.0,
+      "steps": 10,
+      "success": 0
+    },
+    {
+      "episode": 33,
+      "seed": 32,
+      "reward": 0.0,
+      "steps": 46,
+      "success": 0
+    },
+    {
+      "episode": 34,
+      "seed": 33,
+      "reward": 0.0,
+      "steps": 99,
+      "success": 0
+    },
+    {
+      "episode": 35,
+      "seed": 34,
+      "reward": 1.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 36,
+      "seed": 35,
+      "reward": 0.0,
+      "steps": 69,
+      "success": 0
+    },
+    {
+      "episode": 37,
+      "seed": 36,
+      "reward": 1.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 38,
+      "seed": 37,
+      "reward": 0.0,
+      "steps": 30,
+      "success": 0
+    },
+    {
+      "episode": 39,
+      "seed": 38,
+      "reward": 1.0,
+      "steps": 27,
+      "success": 1
+    },
+    {
+      "episode": 40,
+      "seed": 39,
+      "reward": 1.0,
+      "steps": 26,
+      "success": 1
+    },
+    {
+      "episode": 41,
+      "seed": 40,
+      "reward": 1.0,
+      "steps": 9,
+      "success": 1
+    },
+    {
+      "episode": 42,
+      "seed": 41,
+      "reward": 1.0,
+      "steps": 37,
+      "success": 1
+    },
+    {
+      "episode": 43,
+      "seed": 42,
+      "reward": 1.0,
+      "steps": 7,
+      "success": 1
+    },
+    {
+      "episode": 44,
+      "seed": 43,
+      "reward": 1.0,
+      "steps": 47,
+      "success": 1
+    },
+    {
+      "episode": 45,
+      "seed": 44,
+      "reward": 1.0,
+      "steps": 48,
+      "success": 1
+    },
+    {
+      "episode": 46,
+      "seed": 45,
+      "reward": 1.0,
+      "steps": 53,
+      "success": 1
+    },
+    {
+      "episode": 47,
+      "seed": 46,
+      "reward": 1.0,
+      "steps": 27,
+      "success": 1
+    },
+    {
+      "episode": 48,
+      "seed": 47,
+      "reward": 1.0,
+      "steps": 27,
+      "success": 1
+    },
+    {
+      "episode": 49,
+      "seed": 48,
+      "reward": 1.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 50,
+      "seed": 49,
+      "reward": 1.0,
+      "steps": 90,
+      "success": 1
+    },
+    {
+      "episode": 51,
+      "seed": 50,
+      "reward": 1.0,
+      "steps": 48,
+      "success": 1
+    },
+    {
+      "episode": 52,
+      "seed": 51,
+      "reward": 0.0,
+      "steps": 17,
+      "success": 0
+    },
+    {
+      "episode": 53,
+      "seed": 52,
+      "reward": 1.0,
+      "steps": 88,
+      "success": 1
+    },
+    {
+      "episode": 54,
+      "seed": 53,
+      "reward": 1.0,
+      "steps": 25,
+      "success": 1
+    },
+    {
+      "episode": 55,
+      "seed": 54,
+      "reward": 0.0,
+      "steps": 25,
+      "success": 0
+    },
+    {
+      "episode": 56,
+      "seed": 55,
+      "reward": 1.0,
+      "steps": 18,
+      "success": 1
+    },
+    {
+      "episode": 57,
+      "seed": 56,
+      "reward": 1.0,
+      "steps": 46,
+      "success": 1
+    },
+    {
+      "episode": 58,
+      "seed": 57,
+      "reward": 1.0,
+      "steps": 29,
+      "success": 1
+    },
+    {
+      "episode": 59,
+      "seed": 58,
+      "reward": 1.0,
+      "steps": 43,
+      "success": 1
+    },
+    {
+      "episode": 60,
+      "seed": 59,
+      "reward": 1.0,
+      "steps": 46,
+      "success": 1
+    },
+    {
+      "episode": 61,
+      "seed": 60,
+      "reward": 0.0,
+      "steps": 99,
+      "success": 0
+    },
+    {
+      "episode": 62,
+      "seed": 61,
+      "reward": 1.0,
+      "steps": 22,
+      "success": 1
+    },
+    {
+      "episode": 63,
+      "seed": 62,
+      "reward": 1.0,
+      "steps": 7,
+      "success": 1
+    },
+    {
+      "episode": 64,
+      "seed": 63,
+      "reward": 1.0,
+      "steps": 29,
+      "success": 1
+    },
+    {
+      "episode": 65,
+      "seed": 64,
+      "reward": 0.0,
+      "steps": 26,
+      "success": 0
+    },
+    {
+      "episode": 66,
+      "seed": 65,
+      "reward": 1.0,
+      "steps": 26,
+      "success": 1
+    },
+    {
+      "episode": 67,
+      "seed": 66,
+      "reward": 1.0,
+      "steps": 33,
+      "success": 1
+    },
+    {
+      "episode": 68,
+      "seed": 67,
+      "reward": 1.0,
+      "steps": 55,
+      "success": 1
+    },
+    {
+      "episode": 69,
+      "seed": 68,
+      "reward": 1.0,
+      "steps": 19,
+      "success": 1
+    },
+    {
+      "episode": 70,
+      "seed": 69,
+      "reward": 1.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 71,
+      "seed": 70,
+      "reward": 1.0,
+      "steps": 84,
+      "success": 1
+    },
+    {
+      "episode": 72,
+      "seed": 71,
+      "reward": 0.0,
+      "steps": 42,
+      "success": 0
+    },
+    {
+      "episode": 73,
+      "seed": 72,
+      "reward": 0.0,
+      "steps": 62,
+      "success": 0
+    },
+    {
+      "episode": 74,
+      "seed": 73,
+      "reward": 0.0,
+      "steps": 99,
+      "success": 0
+    },
+    {
+      "episode": 75,
+      "seed": 74,
+      "reward": 1.0,
+      "steps": 39,
+      "success": 1
+    },
+    {
+      "episode": 76,
+      "seed": 75,
+      "reward": 1.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 77,
+      "seed": 76,
+      "reward": 1.0,
+      "steps": 21,
+      "success": 1
+    },
+    {
+      "episode": 78,
+      "seed": 77,
+      "reward": 0.0,
+      "steps": 20,
+      "success": 0
+    },
+    {
+      "episode": 79,
+      "seed": 78,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 80,
+      "seed": 79,
+      "reward": 1.0,
+      "steps": 23,
+      "success": 1
+    },
+    {
+      "episode": 81,
+      "seed": 80,
+      "reward": 0.0,
+      "steps": 12,
+      "success": 0
+    },
+    {
+      "episode": 82,
+      "seed": 81,
+      "reward": 1.0,
+      "steps": 58,
+      "success": 1
+    },
+    {
+      "episode": 83,
+      "seed": 82,
+      "reward": 1.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 84,
+      "seed": 83,
+      "reward": 1.0,
+      "steps": 31,
+      "success": 1
+    },
+    {
+      "episode": 85,
+      "seed": 84,
+      "reward": 0.0,
+      "steps": 11,
+      "success": 0
+    },
+    {
+      "episode": 86,
+      "seed": 85,
+      "reward": 0.0,
+      "steps": 73,
+      "success": 0
+    },
+    {
+      "episode": 87,
+      "seed": 86,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 88,
+      "seed": 87,
+      "reward": 0.0,
+      "steps": 99,
+      "success": 0
+    },
+    {
+      "episode": 89,
+      "seed": 88,
+      "reward": 1.0,
+      "steps": 82,
+      "success": 1
+    },
+    {
+      "episode": 90,
+      "seed": 89,
+      "reward": 0.0,
+      "steps": 99,
+      "success": 0
+    },
+    {
+      "episode": 91,
+      "seed": 90,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 92,
+      "seed": 91,
+      "reward": 0.0,
+      "steps": 72,
+      "success": 0
+    },
+    {
+      "episode": 93,
+      "seed": 92,
+      "reward": 1.0,
+      "steps": 59,
+      "success": 1
+    },
+    {
+      "episode": 94,
+      "seed": 93,
+      "reward": 1.0,
+      "steps": 76,
+      "success": 1
+    },
+    {
+      "episode": 95,
+      "seed": 94,
+      "reward": 1.0,
+      "steps": 26,
+      "success": 1
+    },
+    {
+      "episode": 96,
+      "seed": 95,
+      "reward": 1.0,
+      "steps": 43,
+      "success": 1
+    },
+    {
+      "episode": 97,
+      "seed": 96,
+      "reward": 1.0,
+      "steps": 40,
+      "success": 1
+    },
+    {
+      "episode": 98,
+      "seed": 97,
+      "reward": 1.0,
+      "steps": 44,
+      "success": 1
+    },
+    {
+      "episode": 99,
+      "seed": 98,
+      "reward": 1.0,
+      "steps": 47,
+      "success": 1
+    },
+    {
+      "episode": 100,
+      "seed": 99,
+      "reward": 1.0,
+      "steps": 68,
+      "success": 1
+    }
+  ],
+  "evaluation_seconds": 0.04
+}

eval_results/eval_frozenlake8x8.json

@@ -0,0 +1,727 @@
+{
+  "run": "frozenlake8x8",
+  "env_id": "FrozenLake-v1",
+  "env_kwargs": {
+    "map_name": "8x8",
+    "is_slippery": false
+  },
+  "n_training_episodes": 60000,
+  "n_episodes": 100,
+  "mean_reward": 1.0,
+  "std_reward": 0.0,
+  "min_reward": 1.0,
+  "max_reward": 1.0,
+  "success_rate": 1.0,
+  "mean_steps": 14.0,
+  "policy": "greedy (argmax Q)",
+  "model_file": "models\\qtable_frozenlake8x8.pkl",
+  "qtable_shape": [
+    64,
+    4
+  ],
+  "qtable_nonzero": 179,
+  "evaluated_at": "2026-10-30T22:47:24",
+  "episodes": [
+    {
+      "episode": 1,
+      "seed": 0,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 2,
+      "seed": 1,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 3,
+      "seed": 2,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 4,
+      "seed": 3,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 5,
+      "seed": 4,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 6,
+      "seed": 5,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 7,
+      "seed": 6,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 8,
+      "seed": 7,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 9,
+      "seed": 8,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 10,
+      "seed": 9,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 11,
+      "seed": 10,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 12,
+      "seed": 11,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 13,
+      "seed": 12,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 14,
+      "seed": 13,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 15,
+      "seed": 14,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 16,
+      "seed": 15,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 17,
+      "seed": 16,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 18,
+      "seed": 17,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 19,
+      "seed": 18,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 20,
+      "seed": 19,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 21,
+      "seed": 20,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 22,
+      "seed": 21,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 23,
+      "seed": 22,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 24,
+      "seed": 23,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 25,
+      "seed": 24,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 26,
+      "seed": 25,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 27,
+      "seed": 26,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 28,
+      "seed": 27,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 29,
+      "seed": 28,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 30,
+      "seed": 29,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 31,
+      "seed": 30,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 32,
+      "seed": 31,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 33,
+      "seed": 32,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 34,
+      "seed": 33,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 35,
+      "seed": 34,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 36,
+      "seed": 35,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 37,
+      "seed": 36,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 38,
+      "seed": 37,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 39,
+      "seed": 38,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 40,
+      "seed": 39,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 41,
+      "seed": 40,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 42,
+      "seed": 41,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 43,
+      "seed": 42,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 44,
+      "seed": 43,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 45,
+      "seed": 44,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 46,
+      "seed": 45,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 47,
+      "seed": 46,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 48,
+      "seed": 47,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 49,
+      "seed": 48,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 50,
+      "seed": 49,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 51,
+      "seed": 50,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 52,
+      "seed": 51,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 53,
+      "seed": 52,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 54,
+      "seed": 53,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 55,
+      "seed": 54,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 56,
+      "seed": 55,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 57,
+      "seed": 56,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 58,
+      "seed": 57,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 59,
+      "seed": 58,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 60,
+      "seed": 59,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 61,
+      "seed": 60,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 62,
+      "seed": 61,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 63,
+      "seed": 62,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 64,
+      "seed": 63,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 65,
+      "seed": 64,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 66,
+      "seed": 65,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 67,
+      "seed": 66,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 68,
+      "seed": 67,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 69,
+      "seed": 68,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 70,
+      "seed": 69,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 71,
+      "seed": 70,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 72,
+      "seed": 71,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 73,
+      "seed": 72,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 74,
+      "seed": 73,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 75,
+      "seed": 74,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 76,
+      "seed": 75,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 77,
+      "seed": 76,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 78,
+      "seed": 77,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 79,
+      "seed": 78,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 80,
+      "seed": 79,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 81,
+      "seed": 80,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 82,
+      "seed": 81,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 83,
+      "seed": 82,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 84,
+      "seed": 83,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 85,
+      "seed": 84,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 86,
+      "seed": 85,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 87,
+      "seed": 86,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 88,
+      "seed": 87,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 89,
+      "seed": 88,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 90,
+      "seed": 89,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 91,
+      "seed": 90,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 92,
+      "seed": 91,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 93,
+      "seed": 92,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 94,
+      "seed": 93,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 95,
+      "seed": 94,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 96,
+      "seed": 95,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 97,
+      "seed": 96,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 98,
+      "seed": 97,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 99,
+      "seed": 98,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 100,
+      "seed": 99,
+      "reward": 1.0,
+      "steps": 14,
+      "success": 1
+    }
+  ],
+  "evaluation_seconds": 0.02
+}

eval_results/eval_taxi.json

@@ -0,0 +1,724 @@
+{
+  "run": "taxi",
+  "env_id": "Taxi-v3",
+  "env_kwargs": {},
+  "n_training_episodes": 25000,
+  "n_episodes": 100,
+  "mean_reward": 7.56,
+  "std_reward": 2.706732347314747,
+  "min_reward": 3.0,
+  "max_reward": 15.0,
+  "success_rate": 1.0,
+  "mean_steps": 13.44,
+  "policy": "greedy (argmax Q)",
+  "model_file": "models\\qtable_taxi.pkl",
+  "qtable_shape": [
+    500,
+    6
+  ],
+  "qtable_nonzero": 2398,
+  "evaluated_at": "2026-10-30T22:47:35",
+  "episodes": [
+    {
+      "episode": 1,
+      "seed": 16,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 2,
+      "seed": 54,
+      "reward": 12.0,
+      "steps": 9,
+      "success": 1
+    },
+    {
+      "episode": 3,
+      "seed": 165,
+      "reward": 12.0,
+      "steps": 9,
+      "success": 1
+    },
+    {
+      "episode": 4,
+      "seed": 177,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 5,
+      "seed": 191,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 6,
+      "seed": 191,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 7,
+      "seed": 120,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 8,
+      "seed": 80,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 9,
+      "seed": 149,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 10,
+      "seed": 178,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 11,
+      "seed": 48,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 12,
+      "seed": 38,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 13,
+      "seed": 6,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 14,
+      "seed": 125,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 15,
+      "seed": 174,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 16,
+      "seed": 73,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 17,
+      "seed": 50,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 18,
+      "seed": 172,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 19,
+      "seed": 100,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 20,
+      "seed": 148,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 21,
+      "seed": 146,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 22,
+      "seed": 6,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 23,
+      "seed": 25,
+      "reward": 3.0,
+      "steps": 18,
+      "success": 1
+    },
+    {
+      "episode": 24,
+      "seed": 40,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 25,
+      "seed": 68,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 26,
+      "seed": 148,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 27,
+      "seed": 49,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 28,
+      "seed": 167,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 29,
+      "seed": 9,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 30,
+      "seed": 97,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 31,
+      "seed": 164,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 32,
+      "seed": 176,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 33,
+      "seed": 61,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 34,
+      "seed": 7,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 35,
+      "seed": 54,
+      "reward": 12.0,
+      "steps": 9,
+      "success": 1
+    },
+    {
+      "episode": 36,
+      "seed": 55,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 37,
+      "seed": 161,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 38,
+      "seed": 131,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 39,
+      "seed": 184,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 40,
+      "seed": 51,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 41,
+      "seed": 170,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 42,
+      "seed": 12,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 43,
+      "seed": 120,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 44,
+      "seed": 113,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 45,
+      "seed": 95,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 46,
+      "seed": 126,
+      "reward": 15.0,
+      "steps": 6,
+      "success": 1
+    },
+    {
+      "episode": 47,
+      "seed": 51,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 48,
+      "seed": 98,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 49,
+      "seed": 36,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 50,
+      "seed": 135,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 51,
+      "seed": 54,
+      "reward": 12.0,
+      "steps": 9,
+      "success": 1
+    },
+    {
+      "episode": 52,
+      "seed": 82,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 53,
+      "seed": 45,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 54,
+      "seed": 95,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 55,
+      "seed": 89,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 56,
+      "seed": 59,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 57,
+      "seed": 95,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 58,
+      "seed": 124,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 59,
+      "seed": 9,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 60,
+      "seed": 113,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 61,
+      "seed": 58,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 62,
+      "seed": 85,
+      "reward": 14.0,
+      "steps": 7,
+      "success": 1
+    },
+    {
+      "episode": 63,
+      "seed": 51,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 64,
+      "seed": 134,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 65,
+      "seed": 121,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 66,
+      "seed": 169,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 67,
+      "seed": 105,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 68,
+      "seed": 21,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 69,
+      "seed": 30,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 70,
+      "seed": 11,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 71,
+      "seed": 50,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 72,
+      "seed": 65,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 73,
+      "seed": 12,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 74,
+      "seed": 43,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 75,
+      "seed": 82,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 76,
+      "seed": 145,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 77,
+      "seed": 152,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 78,
+      "seed": 97,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 79,
+      "seed": 106,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 80,
+      "seed": 55,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 81,
+      "seed": 31,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 82,
+      "seed": 85,
+      "reward": 14.0,
+      "steps": 7,
+      "success": 1
+    },
+    {
+      "episode": 83,
+      "seed": 38,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 84,
+      "seed": 112,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 85,
+      "seed": 102,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 86,
+      "seed": 168,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 87,
+      "seed": 123,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    },
+    {
+      "episode": 88,
+      "seed": 97,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 89,
+      "seed": 21,
+      "reward": 8.0,
+      "steps": 13,
+      "success": 1
+    },
+    {
+      "episode": 90,
+      "seed": 83,
+      "reward": 10.0,
+      "steps": 11,
+      "success": 1
+    },
+    {
+      "episode": 91,
+      "seed": 158,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 92,
+      "seed": 26,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 93,
+      "seed": 80,
+      "reward": 7.0,
+      "steps": 14,
+      "success": 1
+    },
+    {
+      "episode": 94,
+      "seed": 63,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 95,
+      "seed": 5,
+      "reward": 11.0,
+      "steps": 10,
+      "success": 1
+    },
+    {
+      "episode": 96,
+      "seed": 81,
+      "reward": 9.0,
+      "steps": 12,
+      "success": 1
+    },
+    {
+      "episode": 97,
+      "seed": 32,
+      "reward": 3.0,
+      "steps": 18,
+      "success": 1
+    },
+    {
+      "episode": 98,
+      "seed": 11,
+      "reward": 5.0,
+      "steps": 16,
+      "success": 1
+    },
+    {
+      "episode": 99,
+      "seed": 28,
+      "reward": 4.0,
+      "steps": 17,
+      "success": 1
+    },
+    {
+      "episode": 100,
+      "seed": 148,
+      "reward": 6.0,
+      "steps": 15,
+      "success": 1
+    }
+  ],
+  "evaluation_seconds": 0.02
+}

logs/frozenlake4x4/train_console.log

@@ -0,0 +1,79 @@
+==============================================================================
+Q-Learning 训练 · FrozenLake-v1（4x4，is_slippery=False）
+运行名称: frozenlake4x4    开始时间: 2026-10-30 22:29:00
+==============================================================================
+
+[环境自检]
+  env_id             = FrozenLake-v1
+  env_kwargs         = {'map_name': '4x4', 'is_slippery': False}
+  观测空间           = Discrete(16)  -> 16 个状态
+  动作空间           = Discrete(4)  -> 4 个动作
+  一次 reset 的观测  = 0（整数 = 状态编号，4x4 地图按 row*4+col 编码）
+  每回合最大步数     = 99
+
+[Q 表初始化]
+  Q 表形状 = (16, 4)（= (状态数, 动作数)），初始值全为 0
+  需要学习的参数个数 = 64
+
+[超参数]
+  n_training_episodes = 10000
+  learning_rate       = 0.7
+  gamma               = 0.95
+  max_steps           = 99
+  max_epsilon         = 1.0
+  min_epsilon         = 0.05
+  decay_rate          = 0.0005
+  n_eval_episodes     = 100
+  eval_seed           = []（不指定）
+  train_seed          = 0
+
+[开始训练] 22:29:00  每 500 回合打印一行，每 1000 回合评估一次
+  可复现性：每回合 env.reset(seed=0+episode)，同 --seed 重跑分数一致
+[train] ep=     1/10000 | eps=1.000 | 最近100回合平均回报=  0.00 | 成功率=  0.0% | 平均步数= 11.0 | Q非零项=    0 | 用时= 0.00min
+[train] ep=   500/10000 | eps=0.790 | 最近100回合平均回报=  0.06 | 成功率=  6.0% | 平均步数=  7.6 | Q非零项=   35 | 用时= 0.00min
+[train] ep=  1000/10000 | eps=0.626 | 最近100回合平均回报=  0.26 | 成功率= 26.0% | 平均步数=  7.4 | Q非零项=   35 | 用时= 0.00min
+[eval ] ep=  1000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  1500/10000 | eps=0.499 | 最近100回合平均回报=  0.47 | 成功率= 47.0% | 平均步数=  7.1 | Q非零项=   35 | 用时= 0.00min
+[train] ep=  2000/10000 | eps=0.400 | 最近100回合平均回报=  0.54 | 成功率= 54.0% | 平均步数=  7.1 | Q非零项=   35 | 用时= 0.01min
+[eval ] ep=  2000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  2500/10000 | eps=0.322 | 最近100回合平均回报=  0.66 | 成功率= 66.0% | 平均步数=  7.0 | Q非零项=   35 | 用时= 0.01min
+[train] ep=  3000/10000 | eps=0.262 | 最近100回合平均回报=  0.69 | 成功率= 69.0% | 平均步数=  6.7 | Q非零项=   35 | 用时= 0.01min
+[eval ] ep=  3000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  3500/10000 | eps=0.215 | 最近100回合平均回报=  0.77 | 成功率= 77.0% | 平均步数=  6.7 | Q非零项=   35 | 用时= 0.01min
+[train] ep=  4000/10000 | eps=0.179 | 最近100回合平均回报=  0.84 | 成功率= 84.0% | 平均步数=  6.3 | Q非零项=   35 | 用时= 0.01min
+[eval ] ep=  4000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  4500/10000 | eps=0.150 | 最近100回合平均回报=  0.86 | 成功率= 86.0% | 平均步数=  6.5 | Q非零项=   35 | 用时= 0.01min
+[train] ep=  5000/10000 | eps=0.128 | 最近100回合平均回报=  0.83 | 成功率= 83.0% | 平均步数=  6.2 | Q非零项=   35 | 用时= 0.01min
+[eval ] ep=  5000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  5500/10000 | eps=0.111 | 最近100回合平均回报=  0.85 | 成功率= 85.0% | 平均步数=  6.2 | Q非零项=   35 | 用时= 0.01min
+[train] ep=  6000/10000 | eps=0.097 | 最近100回合平均回报=  0.94 | 成功率= 94.0% | 平均步数=  6.3 | Q非零项=   35 | 用时= 0.01min
+[eval ] ep=  6000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  6500/10000 | eps=0.087 | 最近100回合平均回报=  0.92 | 成功率= 92.0% | 平均步数=  6.1 | Q非零项=   35 | 用时= 0.02min
+[train] ep=  7000/10000 | eps=0.079 | 最近100回合平均回报=  0.88 | 成功率= 88.0% | 平均步数=  6.2 | Q非零项=   35 | 用时= 0.02min
+[eval ] ep=  7000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  7500/10000 | eps=0.072 | 最近100回合平均回报=  0.92 | 成功率= 92.0% | 平均步数=  6.1 | Q非零项=   35 | 用时= 0.02min
+[train] ep=  8000/10000 | eps=0.067 | 最近100回合平均回报=  0.88 | 成功率= 88.0% | 平均步数=  6.0 | Q非零项=   35 | 用时= 0.02min
+[eval ] ep=  8000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  8500/10000 | eps=0.064 | 最近100回合平均回报=  0.92 | 成功率= 92.0% | 平均步数=  6.2 | Q非零项=   35 | 用时= 0.02min
+[train] ep=  9000/10000 | eps=0.061 | 最近100回合平均回报=  0.92 | 成功率= 92.0% | 平均步数=  6.0 | Q非零项=   35 | 用时= 0.02min
+[eval ] ep=  9000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  9500/10000 | eps=0.058 | 最近100回合平均回报=  0.92 | 成功率= 92.0% | 平均步数=  6.0 | Q非零项=   35 | 用时= 0.02min
+[train] ep= 10000/10000 | eps=0.056 | 最近100回合平均回报=  0.93 | 成功率= 93.0% | 平均步数=  6.1 | Q非零项=   35 | 用时= 0.02min
+[eval ] ep= 10000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+
+[训练完成] 22:29:01  总用时 = 0.02 分钟
+
+[最终评估] 用贪婪策略跑 100 局（独立于训练）
+  Mean_reward = 1.00 +/- 0.00
+  成功率      = 100%
+  训练途中最好的周期评估：1.00 ± 0.00 @ 第 1000 回合
+  Q 表非零项 = 35 / 64
+  Q 表最大价值 = 1.0000
+
+[输出] 训练曲线 = logs\frozenlake4x4\train_curve.png
+[输出] 进度 CSV = logs\frozenlake4x4\progress.csv
+[输出] 模型文件 = models\qtable_frozenlake4x4.pkl (1.1 KB)
+
+==============================================================================
+完成。下一步：python evaluate_and_record.py --model models/qtable_frozenlake4x4.pkl
+==============================================================================

logs/frozenlake4x4_slippery/train_console.log

@@ -0,0 +1,79 @@
+==============================================================================
+Q-Learning 训练 · FrozenLake-v1（4x4，is_slippery=True）
+运行名称: frozenlake4x4_slippery    开始时间: 2026-10-30 22:29:40
+==============================================================================
+
+[环境自检]
+  env_id             = FrozenLake-v1
+  env_kwargs         = {'map_name': '4x4', 'is_slippery': True}
+  观测空间           = Discrete(16)  -> 16 个状态
+  动作空间           = Discrete(4)  -> 4 个动作
+  一次 reset 的观测  = 0（整数 = 状态编号，4x4 地图按 row*4+col 编码）
+  每回合最大步数     = 99
+
+[Q 表初始化]
+  Q 表形状 = (16, 4)（= (状态数, 动作数)），初始值全为 0
+  需要学习的参数个数 = 64
+
+[超参数]
+  n_training_episodes = 20000
+  learning_rate       = 0.7
+  gamma               = 0.95
+  max_steps           = 99
+  max_epsilon         = 1.0
+  min_epsilon         = 0.1
+  decay_rate          = 0.0003
+  n_eval_episodes     = 100
+  eval_seed           = 官方 100 个种子
+  train_seed          = 0
+
+[开始训练] 22:29:40  每 1000 回合打印一行，每 2000 回合评估一次
+  可复现性：每回合 env.reset(seed=0+episode)，同 --seed 重跑分数一致
+[train] ep=     1/20000 | eps=1.000 | 最近100回合平均回报=  0.00 | 成功率=  0.0% | 平均步数=  2.0 | Q非零项=    0 | 用时= 0.00min
+[train] ep=  1000/20000 | eps=0.767 | 最近100回合平均回报=  0.01 | 成功率=  1.0% | 平均步数=  8.2 | Q非零项=   44 | 用时= 0.00min
+[train] ep=  2000/20000 | eps=0.594 | 最近100回合平均回报=  0.05 | 成功率=  5.0% | 平均步数=  9.6 | Q非零项=   44 | 用时= 0.01min
+[eval ] ep=  2000 | 贪婪策略 20 局: 回报=  0.00 ±  0.00 | 成功率=  0.0%
+[train] ep=  3000/20000 | eps=0.466 | 最近100回合平均回报=  0.04 | 成功率=  4.0% | 平均步数= 10.5 | Q非零项=   44 | 用时= 0.01min
+[train] ep=  4000/20000 | eps=0.371 | 最近100回合平均回报=  0.03 | 成功率=  3.0% | 平均步数= 11.7 | Q非零项=   44 | 用时= 0.01min
+[eval ] ep=  4000 | 贪婪策略 20 局: 回报=  0.10 ±  0.30 | 成功率= 10.0%
+[train] ep=  5000/20000 | eps=0.301 | 最近100回合平均回报=  0.04 | 成功率=  4.0% | 平均步数= 13.2 | Q非零项=   44 | 用时= 0.02min
+[train] ep=  6000/20000 | eps=0.249 | 最近100回合平均回报=  0.06 | 成功率=  6.0% | 平均步数= 13.1 | Q非零项=   44 | 用时= 0.02min
+[eval ] ep=  6000 | 贪婪策略 20 局: 回报=  0.30 ±  0.46 | 成功率= 30.0%
+[train] ep=  7000/20000 | eps=0.210 | 最近100回合平均回报=  0.10 | 成功率= 10.0% | 平均步数= 15.9 | Q非零项=   44 | 用时= 0.02min
+[train] ep=  8000/20000 | eps=0.182 | 最近100回合平均回报=  0.10 | 成功率= 10.0% | 平均步数= 17.3 | Q非零项=   44 | 用时= 0.03min
+[eval ] ep=  8000 | 贪婪策略 20 局: 回报=  0.10 ±  0.30 | 成功率= 10.0%
+[train] ep=  9000/20000 | eps=0.161 | 最近100回合平均回报=  0.09 | 成功率=  9.0% | 平均步数= 17.4 | Q非零项=   44 | 用时= 0.03min
+[train] ep= 10000/20000 | eps=0.145 | 最近100回合平均回报=  0.12 | 成功率= 12.0% | 平均步数= 16.9 | Q非零项=   44 | 用时= 0.04min
+[eval ] ep= 10000 | 贪婪策略 20 局: 回报=  0.65 ±  0.48 | 成功率= 65.0%
+[train] ep= 11000/20000 | eps=0.133 | 最近100回合平均回报=  0.06 | 成功率=  6.0% | 平均步数= 16.5 | Q非零项=   44 | 用时= 0.04min
+[train] ep= 12000/20000 | eps=0.125 | 最近100回合平均回报=  0.09 | 成功率=  9.0% | 平均步数= 17.4 | Q非零项=   44 | 用时= 0.05min
+[eval ] ep= 12000 | 贪婪策略 20 局: 回报=  0.10 ±  0.30 | 成功率= 10.0%
+[train] ep= 13000/20000 | eps=0.118 | 最近100回合平均回报=  0.09 | 成功率=  9.0% | 平均步数= 18.2 | Q非零项=   44 | 用时= 0.05min
+[train] ep= 14000/20000 | eps=0.114 | 最近100回合平均回报=  0.22 | 成功率= 22.0% | 平均步数= 18.9 | Q非零项=   44 | 用时= 0.06min
+[eval ] ep= 14000 | 贪婪策略 20 局: 回报=  0.65 ±  0.48 | 成功率= 65.0%
+[train] ep= 15000/20000 | eps=0.110 | 最近100回合平均回报=  0.16 | 成功率= 16.0% | 平均步数= 19.4 | Q非零项=   44 | 用时= 0.07min
+[train] ep= 16000/20000 | eps=0.107 | 最近100回合平均回报=  0.21 | 成功率= 21.0% | 平均步数= 21.4 | Q非零项=   44 | 用时= 0.07min
+[eval ] ep= 16000 | 贪婪策略 20 局: 回报=  0.55 ±  0.50 | 成功率= 55.0%
+[train] ep= 17000/20000 | eps=0.105 | 最近100回合平均回报=  0.23 | 成功率= 23.0% | 平均步数= 20.5 | Q非零项=   44 | 用时= 0.08min
+[train] ep= 18000/20000 | eps=0.104 | 最近100回合平均回报=  0.24 | 成功率= 24.0% | 平均步数= 19.9 | Q非零项=   44 | 用时= 0.08min
+[eval ] ep= 18000 | 贪婪策略 20 局: 回报=  0.65 ±  0.48 | 成功率= 65.0%
+[train] ep= 19000/20000 | eps=0.103 | 最近100回合平均回报=  0.18 | 成功率= 18.0% | 平均步数= 20.9 | Q非零项=   44 | 用时= 0.09min
+[train] ep= 20000/20000 | eps=0.102 | 最近100回合平均回报=  0.17 | 成功率= 17.0% | 平均步数= 19.4 | Q非零项=   44 | 用时= 0.09min
+[eval ] ep= 20000 | 贪婪策略 20 局: 回报=  0.75 ±  0.43 | 成功率= 75.0%
+
+[训练完成] 22:29:46  总用时 = 0.10 分钟
+
+[最终评估] 用贪婪策略跑 100 局（独立于训练）
+  Mean_reward = 0.74 +/- 0.44
+  成功率      = 74%
+  训练途中最好的周期评估：0.75 ± 0.43 @ 第 20000 回合
+  Q 表非零项 = 44 / 64
+  Q 表最大价值 = 0.8458
+
+[输出] 训练曲线 = logs\frozenlake4x4_slippery\train_curve.png
+[输出] 进度 CSV = logs\frozenlake4x4_slippery\progress.csv
+[输出] 模型文件 = models\qtable_frozenlake4x4_slippery.pkl (1.4 KB)
+
+==============================================================================
+完成。下一步：python evaluate_and_record.py --model models/qtable_frozenlake4x4_slippery.pkl
+==============================================================================

logs/frozenlake8x8/train_console.log

@@ -0,0 +1,79 @@
+==============================================================================
+Q-Learning 训练 · FrozenLake-v1（8x8，is_slippery=False）
+运行名称: frozenlake8x8    开始时间: 2026-10-30 22:30:02
+==============================================================================
+
+[环境自检]
+  env_id             = FrozenLake-v1
+  env_kwargs         = {'map_name': '8x8', 'is_slippery': False}
+  观测空间           = Discrete(64)  -> 64 个状态
+  动作空间           = Discrete(4)  -> 4 个动作
+  一次 reset 的观测  = 0（整数 = 状态编号，8x8 地图按 row*8+col 编码）
+  每回合最大步数     = 200
+
+[Q 表初始化]
+  Q 表形状 = (64, 4)（= (状态数, 动作数)），初始值全为 0
+  需要学习的参数个数 = 256
+
+[超参数]
+  n_training_episodes = 60000
+  learning_rate       = 0.8
+  gamma               = 0.95
+  max_steps           = 200
+  max_epsilon         = 1.0
+  min_epsilon         = 0.3
+  decay_rate          = 3e-05
+  n_eval_episodes     = 100
+  eval_seed           = 官方 100 个种子
+  train_seed          = 0
+
+[开始训练] 22:30:02  每 3000 回合打印一行，每 6000 回合评估一次
+  可复现性：每回合 env.reset(seed=0+episode)，同 --seed 重跑分数一致
+[train] ep=     1/60000 | eps=1.000 | 最近100回合平均回报=  0.00 | 成功率=  0.0% | 平均步数= 42.0 | Q非零项=    0 | 用时= 0.00min
+[train] ep=  3000/60000 | eps=0.940 | 最近100回合平均回报=  0.00 | 成功率=  0.0% | 平均步数= 23.9 | Q非零项=  161 | 用时= 0.02min
+[train] ep=  6000/60000 | eps=0.885 | 最近100回合平均回报=  0.00 | 成功率=  0.0% | 平均步数= 23.6 | Q非零项=  171 | 用时= 0.05min
+[eval ] ep=  6000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep=  9000/60000 | eps=0.834 | 最近100回合平均回报=  0.02 | 成功率=  2.0% | 平均步数= 22.6 | Q非零项=  175 | 用时= 0.06min
+[train] ep= 12000/60000 | eps=0.788 | 最近100回合平均回报=  0.02 | 成功率=  2.0% | 平均步数= 21.0 | Q非零项=  175 | 用时= 0.08min
+[eval ] ep= 12000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 15000/60000 | eps=0.746 | 最近100回合平均回报=  0.01 | 成功率=  1.0% | 平均步数= 18.5 | Q非零项=  175 | 用时= 0.10min
+[train] ep= 18000/60000 | eps=0.708 | 最近100回合平均回报=  0.01 | 成功率=  1.0% | 平均步数= 18.1 | Q非零项=  175 | 用时= 0.11min
+[eval ] ep= 18000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 21000/60000 | eps=0.673 | 最近100回合平均回报=  0.04 | 成功率=  4.0% | 平均步数= 19.2 | Q非零项=  176 | 用时= 0.13min
+[train] ep= 24000/60000 | eps=0.641 | 最近100回合平均回报=  0.03 | 成功率=  3.0% | 平均步数= 18.1 | Q非零项=  178 | 用时= 0.14min
+[eval ] ep= 24000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 27000/60000 | eps=0.611 | 最近100回合平均回报=  0.08 | 成功率=  8.0% | 平均步数= 17.6 | Q非零项=  179 | 用时= 0.16min
+[train] ep= 30000/60000 | eps=0.585 | 最近100回合平均回报=  0.08 | 成功率=  8.0% | 平均步数= 16.5 | Q非零项=  179 | 用时= 0.17min
+[eval ] ep= 30000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 33000/60000 | eps=0.560 | 最近100回合平均回报=  0.10 | 成功率= 10.0% | 平均步数= 17.6 | Q非零项=  179 | 用时= 0.18min
+[train] ep= 36000/60000 | eps=0.538 | 最近100回合平均回报=  0.07 | 成功率=  7.0% | 平均步数= 14.5 | Q非零项=  179 | 用时= 0.20min
+[eval ] ep= 36000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 39000/60000 | eps=0.517 | 最近100回合平均回报=  0.08 | 成功率=  8.0% | 平均步数= 16.6 | Q非零项=  179 | 用时= 0.21min
+[train] ep= 42000/60000 | eps=0.499 | 最近100回合平均回报=  0.16 | 成功率= 16.0% | 平均步数= 16.7 | Q非零项=  179 | 用时= 0.23min
+[eval ] ep= 42000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 45000/60000 | eps=0.481 | 最近100回合平均回报=  0.16 | 成功率= 16.0% | 平均步数= 16.2 | Q非零项=  179 | 用时= 0.24min
+[train] ep= 48000/60000 | eps=0.466 | 最近100回合平均回报=  0.12 | 成功率= 12.0% | 平均步数= 15.4 | Q非零项=  179 | 用时= 0.26min
+[eval ] ep= 48000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 51000/60000 | eps=0.452 | 最近100回合平均回报=  0.14 | 成功率= 14.0% | 平均步数= 15.0 | Q非零项=  179 | 用时= 0.27min
+[train] ep= 54000/60000 | eps=0.439 | 最近100回合平均回报=  0.17 | 成功率= 17.0% | 平均步数= 14.4 | Q非零项=  179 | 用时= 0.28min
+[eval ] ep= 54000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+[train] ep= 57000/60000 | eps=0.427 | 最近100回合平均回报=  0.25 | 成功率= 25.0% | 平均步数= 14.7 | Q非零项=  179 | ��时= 0.30min
+[train] ep= 60000/60000 | eps=0.416 | 最近100回合平均回报=  0.22 | 成功率= 22.0% | 平均步数= 16.1 | Q非零项=  179 | 用时= 0.31min
+[eval ] ep= 60000 | 贪婪策略 20 局: 回报=  1.00 ±  0.00 | 成功率=100.0%
+
+[训练完成] 22:30:20  总用时 = 0.31 分钟
+
+[最终评估] 用贪婪策略跑 100 局（独立于训练）
+  Mean_reward = 1.00 +/- 0.00
+  成功率      = 100%
+  训练途中最好的周期评估：1.00 ± 0.00 @ 第 6000 回合
+  Q 表非零项 = 179 / 256
+  Q 表最大价值 = 1.0000
+
+[输出] 训练曲线 = logs\frozenlake8x8\train_curve.png
+[输出] 进度 CSV = logs\frozenlake8x8\progress.csv
+[输出] 模型文件 = models\qtable_frozenlake8x8.pkl (2.8 KB)
+
+==============================================================================
+完成。下一步：python evaluate_and_record.py --model models/qtable_frozenlake8x8.pkl
+==============================================================================

logs/taxi/train_console.log

@@ -0,0 +1,84 @@
+==============================================================================
+Q-Learning 训练 · Taxi-v3
+运行名称: taxi    开始时间: 2026-10-30 22:45:31
+==============================================================================
+
+[环境自检]
+  env_id             = Taxi-v3
+  env_kwargs         = {}
+  观测空间           = Discrete(500)  -> 500 个状态
+  动作空间           = Discrete(6)  -> 6 个动作
+  一次 reset 的观测  = 314（整数 = 状态编号，共 500 个）
+  每回合最大步数     = 99
+
+[Q 表初始化]
+  Q 表形状 = (500, 6)（= (状态数, 动作数)），初始值全为 0
+  需要学习的参数个数 = 3000
+
+[超参数]
+  n_training_episodes = 25000
+  learning_rate       = 0.7
+  gamma               = 0.95
+  max_steps           = 99
+  max_epsilon         = 1.0
+  min_epsilon         = 0.05
+  decay_rate          = 0.005
+  n_eval_episodes     = 100
+  eval_seed           = 官方 100 个种子
+  train_seed          = 0
+
+[开始训练] 22:45:31  每 1000 回合打印一行，每 2500 回合评估一次
+  可复现性：每回合 env.reset(seed=0+episode)，同 --seed 重跑分数一致
+[train] ep=     1/25000 | eps=1.000 | 最近100回合平均回报=-405.00 | 成功率=  0.0% | 平均步数= 99.0 | Q非零项=   29 | 用时= 0.00min
+[train] ep=  1000/25000 | eps=0.056 | 最近100回合平均回报=  4.96 | 成功率=100.0% | 平均步数= 14.1 | Q非零项= 2393 | 用时= 0.01min
+[train] ep=  2000/25000 | eps=0.050 | 最近100回合平均回报=  4.12 | 成功率=100.0% | 平均步数= 14.5 | Q非零项= 2393 | 用时= 0.01min
+[eval ] ep=  2500 | 贪婪策略 20 局: 回报=  7.20 ±  2.93 | 成功率=100.0%
+[train] ep=  3000/25000 | eps=0.050 | 最近100回合平均回报=  5.34 | 成功率=100.0% | 平均步数= 13.8 | Q非零项= 2393 | 用时= 0.02min
+[train] ep=  4000/25000 | eps=0.050 | 最近100回合平均回报=  4.83 | 成功率=100.0% | 平均步数= 14.1 | Q非零项= 2393 | 用时= 0.02min
+[train] ep=  5000/25000 | eps=0.050 | 最近100回合平均回报=  4.12 | 成功率=100.0% | 平均步数= 14.7 | Q非零项= 2393 | 用时= 0.03min
+[eval ] ep=  5000 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep=  6000/25000 | eps=0.050 | 最近100回合平均回报=  4.99 | 成功率=100.0% | 平均步数= 13.8 | Q非零项= 2393 | 用时= 0.03min
+[train] ep=  7000/25000 | eps=0.050 | 最近100回合平均回报=  4.21 | 成功率=100.0% | 平均步数= 14.3 | Q非零项= 2393 | 用时= 0.04min
+[eval ] ep=  7500 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep=  8000/25000 | eps=0.050 | 最近100回合平均回报=  4.63 | 成功率=100.0% | 平均步数= 13.9 | Q非零项= 2393 | 用时= 0.04min
+[train] ep=  9000/25000 | eps=0.050 | 最近100回合平均回报=  5.26 | 成功率=100.0% | 平均步数= 13.8 | Q非零项= 2393 | 用时= 0.05min
+[train] ep= 10000/25000 | eps=0.050 | 最近100回合平均回报=  4.50 | 成功率=100.0% | 平均步数= 14.0 | Q非零项= 2394 | 用时= 0.05min
+[eval ] ep= 10000 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep= 11000/25000 | eps=0.050 | 最近100回合平均回报=  5.30 | 成功率=100.0% | 平均步数= 13.6 | Q非零项= 2394 | 用时= 0.06min
+[train] ep= 12000/25000 | eps=0.050 | 最近100回合平均回报=  5.68 | 成功率=100.0% | 平均步数= 13.9 | Q非零项= 2394 | 用时= 0.06min
+[eval ] ep= 12500 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep= 13000/25000 | eps=0.050 | 最近100回合平均回报=  6.05 | 成功率=100.0% | 平均步数= 13.6 | Q非零项= 2394 | 用时= 0.07min
+[train] ep= 14000/25000 | eps=0.050 | 最近100回合平均回报=  5.30 | 成功率=100.0% | 平均步数= 13.7 | Q非零项= 2394 | 用时= 0.07min
+[train] ep= 15000/25000 | eps=0.050 | 最近100回合平均回报=  5.02 | 成功率=100.0% | 平均步数= 13.9 | Q非零项= 2396 | 用时= 0.07min
+[eval ] ep= 15000 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep= 16000/25000 | eps=0.050 | 最近100回合平均回报=  5.16 | 成功率=100.0% | 平均步数= 13.6 | Q非零项= 2396 | 用时= 0.08min
+[train] ep= 17000/25000 | eps=0.050 | 最近100回合平均回报=  4.64 | 成功率=100.0% | 平均步数= 14.0 | Q非零项= 2396 | 用时= 0.08min
+[eval ] ep= 17500 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep= 18000/25000 | eps=0.050 | 最近100回合平均回报=  4.51 | 成功率=100.0% | 平均步数= 14.2 | Q非零项= 2397 | 用时= 0.09min
+[train] ep= 19000/25000 | eps=0.050 | 最近100回合平均回报=  6.06 | 成功率=100.0% | 平均步数= 13.8 | Q非零项= 2397 | 用时= 0.09min
+[train] ep= 20000/25000 | eps=0.050 | 最近100回合平均回报=  4.73 | 成功率=100.0% | 平均步数= 13.8 | Q非零项= 2398 | 用时= 0.10min
+[eval ] ep= 20000 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep= 21000/25000 | eps=0.050 | 最近100回合平均回报=  4.93 | 成功率=100.0% | 平均步数= 14.1 | Q非零项= 2398 | 用时= 0.10min
+[train] ep= 22000/25000 | eps=0.050 | 最近100回合平均回报=  5.22 | 成功率=100.0% | 平均步数= 14.4 | Q非零项= 2398 | 用时= 0.11min
+[eval ] ep= 22500 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+[train] ep= 23000/25000 | eps=0.050 | 最近100回合平均回报=  5.73 | 成功率=100.0% | 平均步数= 13.5 | Q非零项= 2398 | 用时= 0.11min
+[train] ep= 24000/25000 | eps=0.050 | 最近100回合平均回报=  5.22 | 成功率=100.0% | 平均步数= 14.0 | Q非零项= 2398 | 用时= 0.12min
+[train] ep= 25000/25000 | eps=0.050 | 最近100回合平均回报=  4.43 | 成功率=100.0% | 平均步数= 14.0 | Q非零项= 2398 | 用时= 0.12min
+[eval ] ep= 25000 | 贪婪策略 20 局: 回报=  7.30 ±  2.85 | 成功率=100.0%
+
+[训练完成] 22:45:38  总用时 = 0.12 分钟
+
+[最终评估] 用贪婪策略跑 100 局（独立于训练）
+  Mean_reward = 7.56 +/- 2.71
+  成功率      = 100%
+  训练途中最好的周期评估：7.30 ± 2.85 @ 第 5000 回合
+  Q 表非零项 = 2398 / 3000
+  Q 表最大价值 = 20.0000
+
+[输出] 训练曲线 = logs\taxi\train_curve.png
+[输出] 进度 CSV = logs\taxi\progress.csv
+[输出] 模型文件 = models\qtable_taxi.pkl (24.2 KB)
+
+==============================================================================
+完成。下一步：python evaluate_and_record.py --model models/qtable_taxi.pkl
+==============================================================================

metrics.json

@@ -0,0 +1,58 @@
+{
+  "frozenlake4x4": {
+    "env_id": "FrozenLake-v1",
+    "n_episodes": 100,
+    "mean_reward": 1.0,
+    "std_reward": 0.0,
+    "success_rate": 1.0,
+    "mean_steps": 6.0,
+    "qtable_shape": [
+      16,
+      4
+    ],
+    "qtable_nonzero": 35,
+    "evaluated_at": "2026-10-30T22:46:48"
+  },
+  "frozenlake4x4_slippery": {
+    "env_id": "FrozenLake-v1",
+    "n_episodes": 100,
+    "mean_reward": 0.74,
+    "std_reward": 0.4386342439892262,
+    "success_rate": 0.74,
+    "mean_steps": 39.41,
+    "qtable_shape": [
+      16,
+      4
+    ],
+    "qtable_nonzero": 44,
+    "evaluated_at": "2026-10-30T22:46:59"
+  },
+  "frozenlake8x8": {
+    "env_id": "FrozenLake-v1",
+    "n_episodes": 100,
+    "mean_reward": 1.0,
+    "std_reward": 0.0,
+    "success_rate": 1.0,
+    "mean_steps": 14.0,
+    "qtable_shape": [
+      64,
+      4
+    ],
+    "qtable_nonzero": 179,
+    "evaluated_at": "2026-10-30T22:47:24"
+  },
+  "taxi": {
+    "env_id": "Taxi-v3",
+    "n_episodes": 100,
+    "mean_reward": 7.56,
+    "std_reward": 2.706732347314747,
+    "success_rate": 1.0,
+    "mean_steps": 13.44,
+    "qtable_shape": [
+      500,
+      6
+    ],
+    "qtable_nonzero": 2398,
+    "evaluated_at": "2026-10-30T22:47:35"
+  }
+}

models/qtable_frozenlake4x4.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15783f0650b3ecdbf19a79372c3e8e505de805e775f3f3b3283ce4cd2485b011
+size 1175

models/qtable_frozenlake4x4_slippery.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:355c6ee182cb716493c9f7ea9518986b11b9c211cdd07e5564690ca995b863b5
+size 1386

models/qtable_frozenlake8x8.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56b229bccb1b7ae4b5518dd78e0aae8ededf2b766f7d832393a51f4fca295b09
+size 2914

models/qtable_taxi.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42512445ebaa03b152ca4b9f5dc084f363b6a16c5a9d70db18f084625a0ab390
+size 24775

requirements.txt

@@ -0,0 +1,7 @@
+gymnasium>=1.0
+numpy>=1.24
+imageio>=2.31
+imageio-ffmpeg>=0.4
+Pillow>=9.0
+matplotlib>=3.7
+tqdm>=4.65

videos/replay_frozenlake4x4.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f02fda66c2195fd108a330c70e8da165ff0ed144bedffc7f2a4ab5723824e5e
+size 970642

videos/replay_frozenlake4x4_slippery.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a84a49c1451716a0315a6b2620eec47809bbd98030e2c70978d2af1686e915ac
+size 1963828

videos/replay_frozenlake8x8.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a170d8acedfb336a15dc6c1ba368a22681ef21d46ab59f6f7a5e264822be340
+size 1624987

videos/replay_taxi.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b80e69b55c62c9007c2086b6a651a4f9871a2b9fd7ac5220152aa8af9d815c35
+size 1973514

videos/replay_value_based_all.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bba48c8275925470d1dd4873246b6ed7318a642b4159af1af1224df78f385d7
+size 4782880