arxiv:2611.17167

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Published on Nov 12

· Submitted by

wangzixuan on Nov 13

Zhejiang University

Upvote

Authors:

Zixuan Wang ,

Abstract

A decision model that acts poorly can still teach reinforcement learning policies effectively by judging complete action trajectories, improving success on ALFWorld and WebShop.

Generated by deepseek-ai/DeepSeek-V4.1-Flash

Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.

View arXiv page View PDF Project page GitHub 41 Add to collection

Community

wangzx1210

Paper author Paper submitter about 14 hours ago

Decision models output typed decisions (e.g., a probability over candidate actions), so the same model can act as an Actor (choose actions), a Judge (evaluate actions already taken), or a Teacher (score every action to provide step-level rewards for RL). Prior work mostly evaluates them as actors. We ask: can a model that acts poorly still teach well?

🔍 Key findings

On ALFWorld, we compare 3 decision models and 3 general LLMs. As actors, all six get pulled toward an easier subgoal and skip a prerequisite one.
As judges with the full trajectory, the same models correctly credit the steps that advance that prerequisite.
What separates useful from useless actions is judging whether each step helped, not predicting whether the task will succeed.

⚡ As a teacher, our JEV gives step-level signals to a Qwen2.5-1.5B policy trained with GRPO:

ALFWorld: 72.8% → 82.9%
WebShop: 56.8% → 71.4%

Our takeaway: judging an action after the episode reveals consequences that acting can only anticipate, so a weak actor can still be a strong teacher.

🌐 Project page: https://zjureal.com/decision-agents/
💻 Code: https://github.com/ZJU-REAL/decision-agents

polyarti

about 4 hours ago

Asking whether each action helped, rather than whether the task will succeed, is such a clean way to tell useful steps from useless ones. I like that a model that acts badly can still teach well. Did the judge ever credit steps toward the easier subgoal that made the actor stray?