Spaces:
Runtime error
Runtime error
| # Why CodeReviewEnv Matters for MBRL Research | |
| ## The Gap | |
| Every Model-Based RL benchmark today assumes the world is physical: | |
| MuJoCo (robotic joints), Atari (pixel frames), DMControl (physics simulation). | |
| The transition function f(s,a) β s' is always a deterministic physics engine. | |
| The world model just learns to approximate something with a mathematical ground truth. | |
| Agent benchmarks (AgentBench, WebArena, SWE-bench) go further β they operate on | |
| real-world tasks. But they measure only success/failure. There is no MDP formalism, | |
| no trajectory dataset, no path to learning a transition model. They are evaluation | |
| suites, not RL environments. | |
| ## The Formal Problem: Semantic MDP (S-MDP) | |
| We define a **Semantic Markov Decision Process (S-MDP)** as a tuple: | |
| (S, A, T, R, Ξ³) | |
| Where: | |
| - **S** β semantic state space: structured text + metadata (not R^n) | |
| - **A** β structured action space: typed decisions over semantic entities | |
| - **T** β semantic transition function: T(s, a) β s' where s,s' β S | |
| - T is **not expressible as a closed-form equation** | |
| - T must be **learned from trajectory data** | |
| - **R** β shaped reward: R(s, a, s') β [-1, 1] with trajectory-level components | |
| - **Ξ³** β discount factor: 0.95 (standard) | |
| This is distinct from: | |
| - **POMDPs**: partial observability, not semantic transitions | |
| - **Text games** (Jericho, TextWorld): synthetic game worlds, not real-world tasks | |
| - **LLM agent benchmarks**: measure success/failure, no MDP formalism, no trajectories | |
| - **Standard MBRL benchmarks**: continuous vector state, physics-based transitions | |
| ## What Changes Here | |
| In CodeReviewEnv, the transition depends on **meaning**: | |
| - **State**: a pull request with code diffs, bug patterns, author context, review queue | |
| - **Action**: a review decision (label severity, prioritize queue, add comment) | |
| - **Next state**: updated queue, new PRs, changed review context | |
| There is no equation for this. You cannot derive T analytically. | |
| The transition is **semantic** β it depends on understanding code, | |
| recognizing bug patterns, and making judgment calls. | |
| ## The New Model Class: Semantic World Model | |
| A semantic world model must learn: | |
| M(state_t, action_t) β (state_{t+1}, reward_t) | |
| Where state is **structured text**, not a vector. This requires encoding | |
| semantic relationships β bug severity, code quality, reviewer judgment β | |
| into a learnable transition function. | |
| This model class does not exist in the literature. No benchmark supports it. | |
| CodeReviewEnv is the first environment designed for its study. | |
| ## How to Use This Env for MBRL Research | |
| 1. **Collect trajectories**: Run agents to generate (s, a, r, s') data via `export_trajectory()` | |
| 2. **Build dataset**: Load JSONL files into `SemanticTransitionDataset` | |
| 3. **Encode states**: Use sentence-transformers or fine-tuned LLM encoder | |
| 4. **Train transition model**: Fine-tuned encoder + MLP head for (next_state, reward) | |
| 5. **Plan with model**: Imagine rollouts without real env | |
| 6. **Dyna-Q over language**: Sample-efficient learning for knowledge-work agents | |
| ## Key Research Questions | |
| 1. **Error compounding**: Does model error grow exponentially with rollout horizon H | |
| in semantic spaces, as it does in continuous spaces? Or does the structured nature | |
| of text provide error correction? | |
| 2. **State representation**: What encoding produces the best transition model? | |
| Sentence embeddings? Fine-tuned LLM hidden states? Structured feature vectors? | |
| 3. **Transfer**: Can a world model trained on code review transfer to other | |
| knowledge-work domains (email triage, document review, bug prioritization)? | |
| 4. **Planning horizon**: How far ahead can a semantic world model reliably plan? | |
| Is H=3 useful? H=10? Does the answer differ from physical world models? | |
| ## Why This Is Novel | |
| | Benchmark | Domain | State Space | Transition | World Model? | | |
| |-----------|--------|-------------|------------|-------------| | |
| | MuJoCo | Robotics | R^n (joints) | Physics | Yes (Dreamer, MBPO) | | |
| | Atari | Games | Pixels | Engine | Yes (MuZero) | | |
| | DMControl | Physics | R^n | Simulation | Yes (DreamerV3) | | |
| | AgentBench | Tasks | Text | N/A | No (eval only) | | |
| | WebArena | Web | DOM | N/A | No (eval only) | | |
| | SWE-bench | Code | Text | N/A | No (eval only) | | |
| | **CodeReviewEnv** | **Code Review** | **Semantic Text** | **Semantic** | **Yes (this work)** | | |
| CodeReviewEnv is the first benchmark designed for semantic MBRL from the ground up. | |