Spaces:
Runtime error
Runtime error
File size: 4,446 Bytes
dbf5e96 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | # Why CodeReviewEnv Matters for MBRL Research
## The Gap
Every Model-Based RL benchmark today assumes the world is physical:
MuJoCo (robotic joints), Atari (pixel frames), DMControl (physics simulation).
The transition function f(s,a) β s' is always a deterministic physics engine.
The world model just learns to approximate something with a mathematical ground truth.
Agent benchmarks (AgentBench, WebArena, SWE-bench) go further β they operate on
real-world tasks. But they measure only success/failure. There is no MDP formalism,
no trajectory dataset, no path to learning a transition model. They are evaluation
suites, not RL environments.
## The Formal Problem: Semantic MDP (S-MDP)
We define a **Semantic Markov Decision Process (S-MDP)** as a tuple:
(S, A, T, R, Ξ³)
Where:
- **S** β semantic state space: structured text + metadata (not R^n)
- **A** β structured action space: typed decisions over semantic entities
- **T** β semantic transition function: T(s, a) β s' where s,s' β S
- T is **not expressible as a closed-form equation**
- T must be **learned from trajectory data**
- **R** β shaped reward: R(s, a, s') β [-1, 1] with trajectory-level components
- **Ξ³** β discount factor: 0.95 (standard)
This is distinct from:
- **POMDPs**: partial observability, not semantic transitions
- **Text games** (Jericho, TextWorld): synthetic game worlds, not real-world tasks
- **LLM agent benchmarks**: measure success/failure, no MDP formalism, no trajectories
- **Standard MBRL benchmarks**: continuous vector state, physics-based transitions
## What Changes Here
In CodeReviewEnv, the transition depends on **meaning**:
- **State**: a pull request with code diffs, bug patterns, author context, review queue
- **Action**: a review decision (label severity, prioritize queue, add comment)
- **Next state**: updated queue, new PRs, changed review context
There is no equation for this. You cannot derive T analytically.
The transition is **semantic** β it depends on understanding code,
recognizing bug patterns, and making judgment calls.
## The New Model Class: Semantic World Model
A semantic world model must learn:
M(state_t, action_t) β (state_{t+1}, reward_t)
Where state is **structured text**, not a vector. This requires encoding
semantic relationships β bug severity, code quality, reviewer judgment β
into a learnable transition function.
This model class does not exist in the literature. No benchmark supports it.
CodeReviewEnv is the first environment designed for its study.
## How to Use This Env for MBRL Research
1. **Collect trajectories**: Run agents to generate (s, a, r, s') data via `export_trajectory()`
2. **Build dataset**: Load JSONL files into `SemanticTransitionDataset`
3. **Encode states**: Use sentence-transformers or fine-tuned LLM encoder
4. **Train transition model**: Fine-tuned encoder + MLP head for (next_state, reward)
5. **Plan with model**: Imagine rollouts without real env
6. **Dyna-Q over language**: Sample-efficient learning for knowledge-work agents
## Key Research Questions
1. **Error compounding**: Does model error grow exponentially with rollout horizon H
in semantic spaces, as it does in continuous spaces? Or does the structured nature
of text provide error correction?
2. **State representation**: What encoding produces the best transition model?
Sentence embeddings? Fine-tuned LLM hidden states? Structured feature vectors?
3. **Transfer**: Can a world model trained on code review transfer to other
knowledge-work domains (email triage, document review, bug prioritization)?
4. **Planning horizon**: How far ahead can a semantic world model reliably plan?
Is H=3 useful? H=10? Does the answer differ from physical world models?
## Why This Is Novel
| Benchmark | Domain | State Space | Transition | World Model? |
|-----------|--------|-------------|------------|-------------|
| MuJoCo | Robotics | R^n (joints) | Physics | Yes (Dreamer, MBPO) |
| Atari | Games | Pixels | Engine | Yes (MuZero) |
| DMControl | Physics | R^n | Simulation | Yes (DreamerV3) |
| AgentBench | Tasks | Text | N/A | No (eval only) |
| WebArena | Web | DOM | N/A | No (eval only) |
| SWE-bench | Code | Text | N/A | No (eval only) |
| **CodeReviewEnv** | **Code Review** | **Semantic Text** | **Semantic** | **Yes (this work)** |
CodeReviewEnv is the first benchmark designed for semantic MBRL from the ground up.
|