README.md

metadata

title: CodeReviewEnv
emoji: 🔍
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
short_description: RL benchmark for automated code review
tags:
  - openenv
  - reinforcement-learning
  - code-review
  - mbrl
  - semantic-mdp
  - llm-agents

CodeReviewEnv

An OpenEnv-compliant RL environment for automated code review.

CodeReviewEnv formalizes software code review as a Semantic Markov Decision Process (S-MDP). Agents receive buggy code snippets, identify bugs, flag line numbers, suggest fixes, and write review comments. A 5-signal shaped reward provides rich training signal for reinforcement learning.

Why Code Review as RL?

Code review is a core knowledge-work task that requires understanding program semantics, identifying patterns across languages, and producing actionable feedback. Unlike toy text environments, CodeReviewEnv uses real code patterns and deterministic grading --- no LLM-in-the-loop evaluation.

Benchmark	Domain	State Space	Reward	World Model?
MuJoCo	Robotics	R^n	Physics	Yes
Atari	Games	Pixels	Score	Yes
SWE-bench	Code	Text	Pass/Fail	No
CodeReviewEnv	Code Review	Source code	5-signal shaped	Yes

MDP Formulation

Component	Description
State (S)	Buggy source code + language + difficulty metadata
Action (A)	{issues, flagged_lines, suggestion, comment}
Transition (T)	Deterministic: episode ends after review submission
Reward (R)	Weighted combination of 5 normalized signals
Discount	1.0 (single-step episodes)

Reward Breakdown

Signal	Weight	Description
bug_detection	0.40	Fraction of gold bugs identified via keyword matching
fix_quality	0.25	Similarity of suggested fix to gold fix (difflib + keyword overlap)
line_precision	0.15	F1 score of flagged lines vs gold lines (+-3 line tolerance)
comment_quality	0.10	Length, actionability keywords, specificity heuristics
efficiency	0.10	Penalty for excessive steps and hint usage

Hard difficulty gives a 1.5x multiplier on bug_detection.

Difficulty Tiers

Tier	Bugs	Snippet Size	Bug Types
easy	1	< 20 lines	Common patterns
medium	1-2	20-50 lines	Logic errors, mixed
hard	2-3	50+ lines	Interacting bugs, subtle

Procedural Generation

Every reset() produces a unique episode:

1. Picks a clean snippet from a bank of 35+ functions (Python/JS/Go)
2. Applies 1-3 bug injectors based on difficulty
3. Stores gold answers in State (never exposed to agent)

Bug Injectors

Injector	What it does
off_by_one	Flips < to <=, adjusts range() bounds
null_deref	Removes a null/None/nil guard
wrong_operator	Swaps + / -, * / /
unused_var	Inserts dead variable shadowing a live one
logic_inversion	Flips True/False, and/or, ==/!=

Quick Start

# Install
cd code-review-env
pip install -r requirements.txt

# Run baseline (no API key needed)
python baseline.py

# Run tests
pytest tests/ -v

# Start server
uvicorn server.app:app --host 0.0.0.0 --port 7860

Usage

from server.code_review_environment import CodeReviewEnvironment
from models import CodeReviewAction

env = CodeReviewEnvironment()
obs = env.reset(seed=42, difficulty="easy")

print(obs.code)        # Buggy code to review
print(obs.language)    # python, javascript, or go
print(obs.difficulty)  # easy, medium, or hard

action = CodeReviewAction(
    issues=["Off-by-one error in loop boundary"],
    flagged_lines=[3],
    suggestion="Change < to <= on line 3",
    comment="The loop exits too early, missing the last element.",
)

result = env.step(action)
print(f"Reward: {result.reward:.3f}")
print(f"Breakdown: {result.reward_breakdown}")

MCP Tools

For tool-calling agents, CodeReviewEnv exposes MCP endpoints:

POST /mcp/reset            - Start a new episode
POST /mcp/get_code_snippet - Get current buggy code
POST /mcp/submit_review    - Submit review, get reward
POST /mcp/request_hint     - Get a hint (-0.05 reward penalty)
GET  /mcp/get_state        - Get episode state summary

Docker

docker build -t code-review-env .
docker run -p 7860:7860 code-review-env

Inference

export API_BASE_URL="https://openrouter.ai/api/v1"
export API_KEY="your-key"
export MODEL_NAME="openai/gpt-4o-mini"
python inference.py

Project Structure

code-review-env/
  models.py                  # Pydantic Action/Observation/State
  snippet_bank.py            # 35+ snippets + 5 bug injectors
  reward.py                  # 5-signal shaped reward
  client.py                  # EnvClient for WebSocket
  inference.py               # LLM evaluation (mandatory)
  baseline.py                # Heuristic agent (no API key)
  server/
    app.py                   # FastAPI + MCP endpoints
    code_review_environment.py  # Core Environment class
  tests/
    test_code_review_env.py
  openenv.yaml               # OpenEnv manifest
  Dockerfile                 # Production container
  requirements.txt
  README.md

Citation

@software{codereviewenv2024,
  title={CodeReviewEnv: A Semantic MDP for Automated Code Review},
  author={CodeReviewEnv Team},
  year={2024},
  url={https://github.com/ragavrida/code-review-env}
}