Spaces:
Runtime error
Runtime error
| title: CodeReviewEnv | |
| emoji: "\U0001F50D" | |
| colorFrom: blue | |
| colorTo: indigo | |
| sdk: docker | |
| pinned: false | |
| short_description: "RL benchmark for automated code review" | |
| tags: | |
| - openenv | |
| - reinforcement-learning | |
| - code-review | |
| - mbrl | |
| - semantic-mdp | |
| - llm-agents | |
| # CodeReviewEnv | |
| **An OpenEnv-compliant RL environment for automated code review.** | |
| CodeReviewEnv formalizes software code review as a Semantic Markov Decision Process (S-MDP). Agents receive buggy code snippets, identify bugs, flag line numbers, suggest fixes, and write review comments. A 5-signal shaped reward provides rich training signal for reinforcement learning. | |
| ## Why Code Review as RL? | |
| Code review is a core knowledge-work task that requires understanding program semantics, identifying patterns across languages, and producing actionable feedback. Unlike toy text environments, CodeReviewEnv uses real code patterns and deterministic grading --- no LLM-in-the-loop evaluation. | |
| | Benchmark | Domain | State Space | Reward | World Model? | | |
| |-----------|--------|-------------|--------|--------------| | |
| | MuJoCo | Robotics | R^n | Physics | Yes | | |
| | Atari | Games | Pixels | Score | Yes | | |
| | SWE-bench | Code | Text | Pass/Fail | No | | |
| | **CodeReviewEnv** | **Code Review** | **Source code** | **5-signal shaped** | **Yes** | | |
| ## MDP Formulation | |
| | Component | Description | | |
| |-----------|-------------| | |
| | **State (S)** | Buggy source code + language + difficulty metadata | | |
| | **Action (A)** | `{issues, flagged_lines, suggestion, comment}` | | |
| | **Transition (T)** | Deterministic: episode ends after review submission | | |
| | **Reward (R)** | Weighted combination of 5 normalized signals | | |
| | **Discount** | 1.0 (single-step episodes) | | |
| ## Reward Breakdown | |
| | Signal | Weight | Description | | |
| |--------|--------|-------------| | |
| | `bug_detection` | 0.40 | Fraction of gold bugs identified via keyword matching | | |
| | `fix_quality` | 0.25 | Similarity of suggested fix to gold fix (difflib + keyword overlap) | | |
| | `line_precision` | 0.15 | F1 score of flagged lines vs gold lines (+-3 line tolerance) | | |
| | `comment_quality` | 0.10 | Length, actionability keywords, specificity heuristics | | |
| | `efficiency` | 0.10 | Penalty for excessive steps and hint usage | | |
| Hard difficulty gives a 1.5x multiplier on `bug_detection`. | |
| ## Difficulty Tiers | |
| | Tier | Bugs | Snippet Size | Bug Types | | |
| |------|------|-------------|-----------| | |
| | `easy` | 1 | < 20 lines | Common patterns | | |
| | `medium` | 1-2 | 20-50 lines | Logic errors, mixed | | |
| | `hard` | 2-3 | 50+ lines | Interacting bugs, subtle | | |
| ## Procedural Generation | |
| Every `reset()` produces a unique episode: | |
| 1. Picks a clean snippet from a bank of 35+ functions (Python/JS/Go) | |
| 2. Applies 1-3 bug injectors based on difficulty | |
| 3. Stores gold answers in State (never exposed to agent) | |
| ### Bug Injectors | |
| | Injector | What it does | | |
| |----------|-------------| | |
| | `off_by_one` | Flips `<` to `<=`, adjusts `range()` bounds | | |
| | `null_deref` | Removes a null/None/nil guard | | |
| | `wrong_operator` | Swaps `+` / `-`, `*` / `/` | | |
| | `unused_var` | Inserts dead variable shadowing a live one | | |
| | `logic_inversion` | Flips `True`/`False`, `and`/`or`, `==`/`!=` | | |
| ## Quick Start | |
| ```bash | |
| # Install | |
| cd code-review-env | |
| pip install -r requirements.txt | |
| # Run baseline (no API key needed) | |
| python baseline.py | |
| # Run tests | |
| pytest tests/ -v | |
| # Start server | |
| uvicorn server.app:app --host 0.0.0.0 --port 7860 | |
| ``` | |
| ## Usage | |
| ```python | |
| from server.code_review_environment import CodeReviewEnvironment | |
| from models import CodeReviewAction | |
| env = CodeReviewEnvironment() | |
| obs = env.reset(seed=42, difficulty="easy") | |
| print(obs.code) # Buggy code to review | |
| print(obs.language) # python, javascript, or go | |
| print(obs.difficulty) # easy, medium, or hard | |
| action = CodeReviewAction( | |
| issues=["Off-by-one error in loop boundary"], | |
| flagged_lines=[3], | |
| suggestion="Change < to <= on line 3", | |
| comment="The loop exits too early, missing the last element.", | |
| ) | |
| result = env.step(action) | |
| print(f"Reward: {result.reward:.3f}") | |
| print(f"Breakdown: {result.reward_breakdown}") | |
| ``` | |
| ## MCP Tools | |
| For tool-calling agents, CodeReviewEnv exposes MCP endpoints: | |
| ``` | |
| POST /mcp/reset - Start a new episode | |
| POST /mcp/get_code_snippet - Get current buggy code | |
| POST /mcp/submit_review - Submit review, get reward | |
| POST /mcp/request_hint - Get a hint (-0.05 reward penalty) | |
| GET /mcp/get_state - Get episode state summary | |
| ``` | |
| ## Docker | |
| ```bash | |
| docker build -t code-review-env . | |
| docker run -p 7860:7860 code-review-env | |
| ``` | |
| ## Inference | |
| ```bash | |
| export API_BASE_URL="https://openrouter.ai/api/v1" | |
| export API_KEY="your-key" | |
| export MODEL_NAME="openai/gpt-4o-mini" | |
| python inference.py | |
| ``` | |
| ## Project Structure | |
| ``` | |
| code-review-env/ | |
| models.py # Pydantic Action/Observation/State | |
| snippet_bank.py # 35+ snippets + 5 bug injectors | |
| reward.py # 5-signal shaped reward | |
| client.py # EnvClient for WebSocket | |
| inference.py # LLM evaluation (mandatory) | |
| baseline.py # Heuristic agent (no API key) | |
| server/ | |
| app.py # FastAPI + MCP endpoints | |
| code_review_environment.py # Core Environment class | |
| tests/ | |
| test_code_review_env.py | |
| openenv.yaml # OpenEnv manifest | |
| Dockerfile # Production container | |
| requirements.txt | |
| README.md | |
| ``` | |
| ## Citation | |
| ```bibtex | |
| @software{codereviewenv2024, | |
| title={CodeReviewEnv: A Semantic MDP for Automated Code Review}, | |
| author={CodeReviewEnv Team}, | |
| year={2024}, | |
| url={https://github.com/ragavrida/code-review-env} | |
| } | |
| ``` | |