README.md

metadata

license: apache-2.0
base_model: Qwen/Qwen3.5-9B
tags:
  - minecraft
  - embodied-agent
  - reinforcement-learning
  - curiosity

CuriousGame

中文 | English

中文

CuriousGame 是在 Minecraft 里做"找东西"类任务（走近并正对目标方块）的智能体权重，环境为 MineExplorer。仓库里有两种结构：

A：冻结 VLM 感知 + 小型 MLP 控制器（Cerebellum）

• Qwen3.5-9B 不训练，只做感知：每隔几步对当前画面问两个问题，一是定位当前目标的框（框中心按针孔模型换成水平角 h、竖直角 v），二是"画面里有没有这个目标"的 yes 概率 p。
• MLP（约 10 万参数，2 层 256）读最近 20 步的 (p, h, v, 上一步 camera yaw, pitch)，输出三个动作头：移动组合 15 类、yaw 17 档、pitch 13 档。
• 训练：先在 VPT 动作日志上用 hindsight 目标做 BC，再在 MineExplorer 里用每步稠密进展奖励做 GRPO。好奇心版本另加 16 个记忆输入（看过的 8 个方向、各方向 4 格外的格子是否去过）和新颖性奖励。

A 的三个消融（ablations/）

三者的 GRPO 设置都和 A 相同（25 轮 × 48 组 × 6 回合，每回合最多 400 步，同样的场景、奖励和 MLP 超参），感知节奏也同 A（每 3 步问一次 Qwen，中间沿用）。

• hidden_frozen：隐状态直接进 MLP，Qwen 冻结。MLP 的输入不再是 (p, h, v)，而是 Qwen3.5-9B 的末层隐状态：对当前帧问 A 的定位问题，答案前缀固定为 START（见下文），只做 prefill，取 final norm 之后最后一个 prompt token 的 4096 维向量；再拼上最近 20 步的上一步 camera（40 个数）。先在 A 的 BC 控制器的示范上做 BC（bc/），再 GRPO（grpo_r25/）。
• hidden_joint_lora：结构同 hidden_frozen，Qwen 加 LoRA（r 16，alpha 32），GRPO 时 LoRA 和 MLP 一起更新，梯度经隐状态传进 LoRA。起点是 hidden_frozen/bc/ 的 MLP 和零初始化的 LoRA。
• box_joint_lora：接口同 A（MLP 读 p, h, v），Qwen 加 LoRA（r 16，alpha 32），和 MLP 一起训练。rollout 时定位框按 temperature 1 采样，框的 token 按该步优势做 PPO，MLP 做 GRPO；p 由同一个 LoRA 模型读。起点是 A_cerebellum/bc/ 的 MLP 和零初始化的 LoRA。

B：VLM 直接输出 behavior token

• Qwen3.5-9B + LoRA，看一帧画面和任务文本，输出一个 behavior code（FSQ [8,5,5,5]，共 1000 个码，每个码对应 20 tick 的动作块）。
• 训练：在 A 的 BC 控制器的示范上 SFT，再 GRPO。好奇心版本在 prompt 后加两串 8 位 0/1 记忆，并加新颖性奖励。

English

CuriousGame contains agent weights for "find" tasks in Minecraft: the agent has to walk up to a target block and face it. The environment is MineExplorer. The repo holds two designs:

A: frozen VLM perception + small MLP controller (Cerebellum)

• Qwen3.5-9B stays frozen and only does perception. Every few steps it answers two questions about the current frame:
  • a bounding box for the current target, whose center becomes the horizontal and vertical angles h and v through a pinhole model;
  • the yes-probability p that the target is in view.
• The MLP (about 100k parameters, two hidden layers of 256) reads (p, h, v, previous camera yaw, previous camera pitch) over the last 20 steps. It has three action heads: 15 movement combinations, 17 yaw bins and 13 pitch bins.
• Training is BC on VPT action logs with hindsight goals, then GRPO in MineExplorer with a dense per-step progress reward.
• The curiosity variant adds 16 memory inputs: which of 8 directions have been seen, and whether the cell 4 blocks along each has been visited. It also adds a novelty reward.

Three ablations of A (ablations/)

All three use A's GRPO setup (25 rounds x 48 groups x 6 episodes of at most 400 steps, the same scenes, reward and MLP hyperparameters) and A's perception schedule (Qwen is asked every 3 steps and its output is held in between).

• hidden_frozen: the hidden state goes straight into the MLP and Qwen stays frozen. Instead of (p, h, v), the MLP reads Qwen3.5-9B's last-layer hidden state. Qwen gets A's box question on the current frame with the answer prefix fixed to START (below), runs prefill only, and the feature is the 4096-dim vector at the last prompt token after the final norm. The camera actions of the last 20 steps (40 values) are appended. Training is BC on demonstrations of A's BC controller (bc/), then GRPO (grpo_r25/).
• hidden_joint_lora: the same design as hidden_frozen, with a LoRA on Qwen (r 16, alpha 32). GRPO updates the LoRA and the MLP together, and the gradient reaches the LoRA through the hidden state. It starts from the MLP in hidden_frozen/bc/ and a zero-initialised LoRA.
• box_joint_lora: A's interface (the MLP reads p, h, v), with a LoRA on Qwen (r 16, alpha 32) trained together with the MLP. During rollouts the box is sampled at temperature 1 and its tokens get a PPO update weighted by that step's advantage, while the MLP gets GRPO. p is read from the same LoRA model. It starts from the MLP in A_cerebellum/bc/ and a zero-initialised LoRA.

B: the VLM outputs behavior tokens directly

• Qwen3.5-9B with a LoRA reads one frame and the task text and outputs a behavior code. The codes come from an FSQ [8,5,5,5] tokenizer: 1000 codes, each one a 20-tick action chunk.
• Training is SFT on demonstrations from A's BC controller, then GRPO.
• The curiosity variant appends two 8-bit 0/1 memory strings to the prompt and adds the novelty reward.

Files

Path	Content
Qwen3.5-9B/	Unmodified base Qwen3.5-9B (Apache-2.0), the frozen perception of A and the base of B's LoRAs. 未改动的基座 Qwen3.5-9B。
A_cerebellum/bc/ctrl.npz	A's MLP after BC (camera-history rows of w0 zeroed). A 的 BC 控制器。
A_cerebellum/grpo_r25/ctrl.npz	A's MLP after 25 GRPO rounds. A 的 GRPO 第 25 轮。
A_cerebellum/grpo_r25_curiosity/ctrl.npz	A with curiosity (116 inputs), 25 GRPO rounds. A 好奇心版 GRPO 第 25 轮。
ablations/hidden_frozen/bc/ctrl.npz	Hidden-state MLP after BC; also the starting MLP of hidden_joint_lora. 隐状态 MLP 的 BC 结果，也是 hidden_joint_lora 的起点。
ablations/hidden_frozen/grpo_r25/ctrl.npz	Hidden-state MLP, frozen Qwen, 25 GRPO rounds. 隐状态 MLP，Qwen 冻结，GRPO 第 25 轮。
ablations/hidden_joint_lora/grpo_r25/	Hidden-state MLP (ctrl.npz) and Qwen LoRA, trained together, 25 GRPO rounds. 隐状态 MLP 和 Qwen LoRA 一起训练，GRPO 第 25 轮。
ablations/box_joint_lora/grpo_r25/	A's MLP (ctrl.npz) and the box LoRA, trained together, 25 GRPO rounds. A 的 MLP 和定位框 LoRA 一起训练，GRPO 第 25 轮。
B_behavior_token/sft/	B's LoRA after SFT. B 的 SFT LoRA。
B_behavior_token/grpo_r25/	B's LoRA after 25 GRPO rounds. B 的 GRPO 第 25 轮。
B_behavior_token/grpo_r25_curiosity/	B with curiosity, 25 GRPO rounds. B 好奇心版 GRPO 第 25 轮。
B_behavior_token/tokenizer/	FSQ behavior tokenizer (fsq.pt) and the code -> 20-tick action decode table (table.npz). 行为 tokenizer 和解码表。

MLP format (A)

ctrl.npz holds w0, b0, w1, b1, w2, b2 with weights as (in, out): logits = relu(relu(x @ w0 + b0) @ w1 + b1) @ w2 + b2. The 45 logits split as 15 movement / 17 yaw / 13 pitch. The three heads are sampled independently.

The input x has 100 values: 20 steps, oldest first, 5 values per step:

• p;
• h / 51.2, where h > 0 means right of the view center;
• v / 35, where v > 0 means above it;
• the previous camera action yaw / 64;
• the previous camera action pitch / 16.

The curiosity model appends 16 memory inputs.

The hidden-state MLPs (hidden_frozen, hidden_joint_lora) have the same layout with 4136 inputs: the 4096-dim hidden state (float, no normalisation), then the previous camera action [yaw / 64, pitch / 16] of the last 20 steps, oldest first. With hidden_joint_lora, the hidden state is taken with its LoRA loaded. START = ```json\n[\n\t{"bbox_2d": [.

Bins:

• Yaw (degrees, > 0 turns right): 0, ±0.5, ±1, ±2, ±4, ±8, ±16, ±32, ±64.
• Pitch (degrees, > 0 looks down): 0, ±0.5, ±1, ±2, ±4, ±8, ±16.
• Movement classes: none, forward, forward+jump, forward+right, forward+left, forward+sprint, forward+jump+sprint, back, right, left, back+left, back+right, forward+right+jump, jump, forward+left+jump.

Perception prompts (thinking off, temperature 0, 320x180 frame):

• box: Locate the {target} in this image and report its bbox coordinates in JSON format. If there is no {target} in the image, output an empty list []. (coordinates 0-1000; no box gives h = v = 0)
• presence: Is there a {target} in this image? Answer yes or no. with p = P(yes) / (P(yes) + P(no)).

Prompt (B)

Task: find {target}. Reply with the behavior code for the next second. The curiosity variant appends Seen directions (0, 45, ..., 315 deg right of view): {8 bits}. Visited cells 4 blocks along them: {8 bits}.

推理代码 / Inference code

code/ 是评测时实际用的脚本，原样上传，保留原来的相对目录（脚本之间按相对路径 import）。patches/ 是对 MineExplorer 的两个补丁。

• code/train/ctrl_policy.py：A 和 B 的策略。Policy 是 A（好奇心版按 w0 的行数自动加记忆输入；box_joint_lora 也用它），BTPolicy 是 B，make_policy 按 kind 选择。
• code/train/hid_policy.py、code/train/hid_server.py：hidden_frozen、hidden_joint_lora 的策略和隐状态特征服务（HTTP，--adapter 加载 LoRA）。hid_server.py 的模型加载函数来自 code/train/bt_train.py。
• code/bench/diag_perception*.py：感知问题原文、框的解析和视场常量，ctrl_policy.py 从这里 import；code/train/grpo_locations.json 在 import 时读取。
• code/bench/splits_find25.json：find25 的 25 个任务编号。
• patches/MineExplorer.patch：MineExplorer（commit cca4c5e）的 eval_benchmark.py，加了 MC_POLICY 入口（用上面的策略代替 LLM agent）、--tasks、多个 sandbox 轮流分配（MC_SANDBOX_URLS）；env/minerl_sandbox.py 的 reset 超时可配（MC_RESET_TIMEOUT）。
• patches/sandbox_mc_server.patch：sandbox 镜像里的 minestudio/mc_server.py，step 返回 info（策略要读 player_pos），每次 reset 重新设 world seed。

code/ holds the scripts the evaluations ran, uploaded unchanged in their original relative layout (they import each other by relative path); patches/ holds two patches to MineExplorer.

• code/train/ctrl_policy.py: the policies of A and B. Policy is A (the curiosity variant turns on the memory inputs from the row count of w0; box_joint_lora uses it too), BTPolicy is B, and make_policy picks one by kind.
• code/train/hid_policy.py, code/train/hid_server.py: the policy of hidden_frozen / hidden_joint_lora and the HTTP hidden-state server (--adapter loads a LoRA). hid_server.py takes its model loader from code/train/bt_train.py.
• code/bench/diag_perception*.py: the perception prompts, the box parser and the field-of-view constants, imported by ctrl_policy.py; code/train/grpo_locations.json is read at import time.
• code/bench/splits_find25.json: the 25 task ids of find25.
• patches/MineExplorer.patch: MineExplorer (commit cca4c5e) eval_benchmark.py gains the MC_POLICY hook (runs the policies above instead of the LLM agent), --tasks and round-robin over several sandboxes (MC_SANDBOX_URLS); env/minerl_sandbox.py gets a configurable reset timeout (MC_RESET_TIMEOUT).
• patches/sandbox_mc_server.patch: minestudio/mc_server.py in the sandbox image returns the step info (the policies read player_pos) and re-applies the world seed on every reset.

Setup:

cd code
git clone https://github.com/meituan-longcat/MineExplorer.git
git -C MineExplorer checkout cca4c5e && git -C MineExplorer apply ../../patches/MineExplorer.patch
mkdir -p models && ln -s ../../Qwen3.5-9B models/Qwen3.5-9B

Start the sandbox from MineExplorer's Docker image (davidzhth/mineexplorer:0.0.1, see its README) with patches/sandbox_mc_server.patch applied to /home/sandbox/mc/lumine-client/minestudio/mc_server.py. Versions used: vLLM 0.27.1; torch 2.13, transformers 5.15, peft 0.21 for hid_server.py; Python 3.11 with MineExplorer's requirements plus numpy, openai, requests for the eval.

Perception server:

# A: base Qwen3.5-9B
vllm serve models/Qwen3.5-9B --served-model-name perception --dtype bfloat16 --max-model-len 16384 \
  --limit-mm-per-prompt '{"image": 24}' --port 9060
# B or box_joint_lora: the LoRA under the name the policy asks for (box: ../ablations/box_joint_lora/grpo_r25)
vllm serve models/Qwen3.5-9B --served-model-name perception-base --dtype bfloat16 --max-model-len 16384 \
  --limit-mm-per-prompt '{"image": 24}' --port 9060 --enable-lora --max-lora-rank 16 \
  --lora-modules perception=../B_behavior_token/grpo_r25
# hidden_frozen (no adapter) and hidden_joint_lora
CUDA_VISIBLE_DEVICES=0 python train/hid_server.py --port 9210 --adapter ../ablations/hidden_joint_lora/grpo_r25

Eval (find25, 1000 steps; our tables pool seeds 0, 10000, 20000; keep --num-workers at 2 or more, since 1 takes a serial path that ignores MC_SANDBOX_URLS):

cd MineExplorer
MC_SANDBOX_URLS=http://localhost:8031,http://localhost:8032 MC_RESET_TIMEOUT=1200 \
AGENT_API_KEY=unused AGENT_API_BASE=unused MC_POLICY='<json below>' \
python eval_benchmark.py --model ctrl --benchmark-dir benchmark --output-dir out_s0 \
  --tasks $(python -c "import json; print(','.join(json.load(open('../bench/splits_find25.json'))['eval']))") \
  --num-workers 13 --max-steps 1000

Model	Server	MC_POLICY
A (any A_cerebellum/*/ctrl.npz)	vLLM, base	{"npz": "/abs/ctrl.npz", "perception": "vlm", "url": "http://localhost:9060/v1", "model": "perception", "seed": 0}
box_joint_lora	vLLM + its LoRA	as A, with its ctrl.npz
B	vLLM + B's LoRA	{"kind": "bt", "table": "/abs/table.npz", "url": "http://localhost:9060/v1", "model": "perception", "seed": 0}, plus "memory": true for grpo_r25_curiosity
hidden_frozen, hidden_joint_lora	hid_server.py (adapter only for hidden_joint_lora)	{"kind": "hid", "npz": "/abs/ctrl.npz", "url": "http://localhost:9210", "seed": 0}

Each episode writes steps.jsonl with the policy's per-step log (inputs, sampled bins, perception replies).