final README

README.md

@@ -1,4 +1,5 @@
 ---
+
 title: AI Code Review OpenEnv
 emoji: 🤖
 colorFrom: blue
@@ -6,359 +7,221 @@ colorTo: purple
 sdk: docker
 app_file: inference.py
 pinned: false
----
-# 🔍 AI Code Review — OpenEnv Environment
-
-An RL environment where an AI agent performs structured code review tasks —
-identifying bugs and proposing fixes across progressively harder JavaScript challenges.
+-------------
 
-Built for the **Meta × Hugging Face × OpenEnv × Scaler** hackathon.
-
----
+# 🔍 AI Code Review — OpenEnv RL Environment
 
-## 📌 Project Overview
+> Train and evaluate AI agents on real-world code review using structured RL environments.
 
-Most code review tools either flag syntax errors with a linter or generate
-free-form suggestions with an LLM. Neither approach is structured for
-reinforcement learning — there is no clear action space, no reward signal,
-and no way to train an agent to improve over time.
+A **real-world reinforcement learning environment** where AI agents learn structured code review:
+**identify bugs → reason about them → fix them correctly**.
 
-**AI Code Review** frames code review as an RL problem:
-
-- The **agent** reads buggy code and must first *identify* the bug, then *fix* it
-- The **environment** grades both steps deterministically and returns a shaped reward
-- The **episode** runs three tasks of increasing difficulty, giving the agent a
-  continuous learning signal across syntax, logic, and performance bug categories
-
-This makes it possible to train or evaluate LLM agents on code review skills in
-a reproducible, measurable way — without relying on human judges or expensive
-model-based graders.
+Built for the **Meta × Hugging Face × OpenEnv × Scaler Hackathon 2026 (Round 1 Submission)**.
 
 ---
 
-## 💡 Motivation
+## 🚀 Why This Matters
 
-Code review is one of the highest-leverage activities in a software team.
-Developers spend an estimated **10–15% of their time** reviewing others' code,
-and studies show that poor reviews are a leading source of production bugs.
+Most AI coding tools are built for *generation*, not *review*.
+But in real-world software engineering, that's the wrong priority:
 
-Despite this, most AI coding tools are optimised for *generation*, not *review*.
-There are no widely-adopted RL benchmarks for teaching agents to:
+* Code review is the last line of defence against production bugs
+* Bugs often slip through because reviewers miss the *why*, not the *what*
+* Understanding why code is wrong matters more than just patching it
 
-- Classify the type of bug they are looking at
-- Explain why the code is wrong
-- Produce a minimal, correct fix
+👉 This environment turns **code review into a measurable, learnable RL problem** — with structure, reward signal, and reproducibility built in from the ground up.
 
-This environment fills that gap. It models the two-step workflow a human
-reviewer follows — *"what is wrong?"* then *"here is the fix"* — and rewards
-the agent at each step. The result is a training signal that encourages
-coherent reasoning, not just lucky guesses.
+---
 
-**Real-world applications of agents trained in this environment:**
+## 🧠 What Makes This Environment Strong
 
-- Automated PR review bots that flag and fix common bug patterns
-- CI/CD pipeline integration to catch issues before human review
-- Developer tooling that explains *why* code is wrong, not just that it is
-- Fine-tuning data generation for code-focused LLMs
+| Property                  | Detail                                                 |
+| ------------------------- | ------------------------------------------------------ |
+| ✅ Real-world task         | Code review, not a toy problem                         |
+| ✅ Two-phase reasoning     | Agent must identify before it can fix                  |
+| ✅ Deterministic grading   | No LLM judge, no randomness                            |
+| ✅ Reward shaping          | Partial progress is rewarded at every step             |
+| ✅ Reproducible evaluation | Same input always produces same score                  |
+| ✅ Spec compliant          | Fully compatible with OpenEnv + hackathon requirements |
 
 ---
 
 ## 🏗️ Environment Design
 
-The environment follows the [OpenEnv](https://github.com/meta-pytorch/OpenEnv)
-specification, implementing three required interfaces: `reset()`, `step()`, and
-the `state` property.
+Implements the full OpenEnv interface — three methods, no more:
 
-### Observation Space
+```python
+obs      = env.reset()       # Start a new episode
+result   = env.step(action)  # Execute one action → {observation, reward, done, info}
+metadata = env.state         # Inspect episode state
+```
 
-Each observation exposes the following fields:
+---
+
+## 📥 Observation Space
 
-| Field        | Type            | Description                                                  |
-|--------------|-----------------|--------------------------------------------------------------|
-| `code`       | `str`           | The buggy JavaScript code to review                          |
-| `task`       | `str`           | Full task prompt: difficulty, category, description, phase   |
-| `history`    | `List[str]`     | Log of all prior steps taken in this episode                 |
-| `task_id`    | `int`           | Current task index (1, 2, or 3)                              |
-| `language`   | `str`           | Always `"javascript"` in this environment                    |
-| `difficulty` | `str`           | One of `easy`, `medium`, `hard`                              |
-| `category`   | `str`           | One of `syntax`, `logic`, `performance`                      |
+| Field        | Description                         |
+| ------------ | ----------------------------------- |
+| `code`       | Buggy JavaScript code to review     |
+| `task`       | Task description + current phase    |
+| `history`    | Log of all prior steps this episode |
+| `task_id`    | Task number (1, 2, or 3)            |
+| `language`   | `javascript`                        |
+| `difficulty` | `easy` / `medium` / `hard`          |
+| `category`   | `syntax` / `logic` / `performance`  |
 
-The `task` field changes with phase — in the `IDENTIFY` phase it instructs the
-agent to explain the bug; in the `FIX` phase it instructs the agent to correct it.
+---
 
-### Action Space
+## 🎮 Action Space
 
-Each step the agent submits one action:
+| Field         | Values                        |
+| ------------- | ----------------------------- |
+| `action_type` | `"identify"` or `"fix"`       |
+| `content`     | Free-text explanation or code |
 
-| Field         | Type  | Values              | Description                                    |
-|---------------|-------|---------------------|------------------------------------------------|
-| `action_type` | `str` | `identify` / `fix`  | Which phase this action addresses              |
-| `content`     | `str` | Free text or code   | Bug explanation (identify) or corrected code (fix) |
+---
 
-The expected episode flow per task is:
+## 🔁 Episode Flow
 
 ```
-identify → fix → [next task] → identify → fix → ...
+identify → fix → identify → fix → identify → fix
+   task 1            task 2            task 3
 ```
 
-Deviating from this order is allowed but penalised (see Reward section).
-
-### State
-
-The `state` property returns an `EpisodeState` dataclass with episode metadata:
-
-| Field                      | Type        | Description                          |
-|----------------------------|-------------|--------------------------------------|
-| `current_task_id`          | `int`       | ID of the task currently active      |
-| `current_task_difficulty`  | `str`       | Difficulty of the current task       |
-| `current_task_category`    | `str`       | Bug category of the current task     |
-| `phase`                    | `str`       | Current phase: `identify` or `fix`   |
-| `step`                     | `int`       | Total steps taken so far             |
-| `total_reward`             | `float`     | Cumulative reward across the episode |
-| `done`                     | `bool`      | Whether the episode has ended        |
-| `history`                  | `List[str]` | Step-by-step reward log              |
+One episode covers all 3 tasks in order.
 
 ---
 
 ## 🧪 Tasks
 
-The environment defines three tasks covering distinct bug categories. Each task
-requires exactly two steps: one `identify` and one `fix`.
-
-### Task 1 — Easy: Missing Closing Parenthesis
-
-**Category:** Syntax  
-**Language:** JavaScript
+### 🟢 Task 1 — Syntax Error (Easy)
 
 ```javascript
-// Buggy code
+// buggy
 function add(a, b {
   return a + b;
 }
 
-// Expected fix
+// fixed
 function add(a, b) {
   return a + b;
 }
 ```
 
-The function parameter list is missing a closing parenthesis `)`. This is a
-parse-time `SyntaxError` — no JavaScript engine will execute this code. The
-agent must identify that the issue is a missing parenthesis and restore it.
-
 ---
 
-### Task 2 — Medium: Wrong Even/Odd Condition
-
-**Category:** Logic  
-**Language:** JavaScript
+### 🟡 Task 2 — Logic Bug (Medium)
 
 ```javascript
-// Buggy code
+// buggy
 function isEven(n) {
   return n % 2 === 1;
 }
 
-// Expected fix
+// fixed
 function isEven(n) {
   return n % 2 === 0;
 }
 ```
 
-The function uses `=== 1` instead of `=== 0` in the remainder check, inverting
-the result. It silently returns `true` for odd numbers and `false` for even ones —
-no crash, just wrong output. The agent must recognise this as a logic error
-and correct the condition.
-
 ---
 
-### Task 3 — Hard: Inefficient Array Iteration
-
-**Category:** Performance  
-**Language:** JavaScript
+### 🔴 Task 3 — Performance Issue (Hard)
 
 ```javascript
-// Buggy code
+// buggy
 for (let i = 0; i < arr.length; i++) {
   console.log(arr[i]);
 }
 
-// Expected fix
+// fixed
 arr.forEach(item => {
   console.log(item);
 });
 ```
 
-The loop re-evaluates `arr.length` on every iteration and uses verbose
-index-based element access. A `forEach` approach is cleaner, avoids repeated
-property lookups, and is idiomatic in modern JavaScript. The agent must
-recognise this as a performance and style issue and propose the functional
-alternative.
-
 ---
 
-## 🎯 Reward Function
-
-Rewards are **fully deterministic** — no LLM judges, no randomness. The grader
-uses keyword matching for identify and code comparison for fix.
+## 🎯 Reward System
 
-> **Deterministic grading ensures reproducibility across runs, which is critical
-> for fair agent evaluation and leaderboard consistency.**
+### ✔ Deterministic & Reproducible
 
-All rewards are strictly clamped to **[0.0, 1.0]** before being returned from
-`step()`, ensuring full compliance with the OpenEnv specification.
+No randomness. Same input → same score.
 
-### Identify Phase
+---
 
-Smooth partial scoring — every matched keyword contributes proportionally
-with no hard threshold. A single correct keyword earns partial credit.
+### 🧩 Identify Reward
 
 ```
-score = matched_keywords / total_keywords   (capped at 1.0)
+score = matched_keywords / total_keywords
 ```
 
-Keywords use **whole-word boundary matching** — `"paren"` does not match inside
-`"parenthesis"`, preventing accidental substring hits. This is fully deterministic
-with no randomness involved.
-
-### Fix Phase
-
-Fix scoring runs four tiers in order:
-
-| Tier | Condition                        | Score |
-|------|----------------------------------|-------|
-| 1    | Exact match (after stripping)    | 1.0   |
-| 2    | Whitespace-normalised match      | 0.9   |
-| 3    | ≥ 80% token overlap              | 0.6   |
-| 4    | ≥ 50% token overlap              | 0.3   |
-| —    | Below 50% overlap                | 0.0   |
-
-The grader automatically extracts code from fenced ` ``` ` blocks if present,
-so agents that wrap their fix in a code block are handled correctly.
-
-### Identify Bonus
-
-When an agent scores `≥ 0.4` on the identify step, a **+0.1 bonus** is applied
-to the fix reward before clamping. This incentivises the agent to reason through
-the bug before jumping to a fix — coherent two-step reasoning earns more than
-lucky fixes.
-
-> **The two-phase (identify → fix) structure mirrors real developer workflows
-> and encourages reasoning before action, reducing reward hacking.**
-
-The identify bonus is always applied before the final clamp, so the maximum
-fix reward remains **1.0** (not 1.1). No reward ever exceeds 1.0.
-
-### Behaviour for Edge Cases
-
-| Situation                                | Reward              |
-|------------------------------------------|---------------------|
-| Skipped identify, fix submitted directly | fix score × 0.5     |
-| Repeated `identify` in fix phase         | 0.0                 |
-| Unknown `action_type`                    | 0.0                 |
+---
 
-### Reward Summary
+### 🛠️ Fix Reward
 
-| Step               | Max Reward |
-|--------------------|------------|
-| Identify (perfect) | 1.0        |
-| Fix (exact + bonus)| 1.0 (clamped from 1.1) |
-| Full episode       | ~6.0 (3 tasks × identify + fix) |
+| Condition             | Score |
+| --------------------- | ----- |
+| Exact match           | 1.0   |
+| Whitespace-normalised | 0.9   |
+| ≥ 80% token overlap   | 0.6   |
+| ≥ 50% token overlap   | 0.3   |
+| Below 50%             | 0.0   |
 
 ---
 
-## ⚙️ Setup Instructions
-
-### Prerequisites
+### 🎁 Identify Bonus
 
-- [Docker](https://docs.docker.com/get-docker/) installed and running
-- A Hugging Face token (`HF_TOKEN`) with inference access
++0.1 bonus if identify ≥ 0.4
+(final reward clamped to 1.0)
 
-### Environment Variables
+---
 
-| Variable      | Required | Default                        | Description                          |
-|---------------|----------|--------------------------------|--------------------------------------|
-| `HF_TOKEN`    | ✅ Yes   | —                              | HF token used as the API key         |
-| `API_BASE_URL`| ❌ No    | `https://api.openai.com/v1`   | OpenAI-compatible inference endpoint |
-| `MODEL_NAME`  | ❌ No    | `gpt-4.1-mini`                | Model name for the LLM client        |
+### ⚠️ Edge Case Handling
 
-### Build the Docker Image
+| Situation              | Effect          |
+| ---------------------- | --------------- |
+| Skipped identify phase | fix score × 0.5 |
+| Unknown `action_type`  | 0.0             |
+| Repeated identify      | 0.0             |
 
-```bash
-docker build -t ai-code-review-env:latest .
-```
+---
 
-### Run the Environment
+### 📊 Max Reward Per Episode
 
-```bash
-docker run --rm \
-  -e HF_TOKEN=your_hf_token_here \
-  ai-code-review-env:latest
-```
+| Scope        | Max Reward |
+| ------------ | ---------- |
+| Per step     | 1.0        |
+| Per task     | ~2.0       |
+| Full episode | ~6.0       |
 
-### Run with a Custom Endpoint
+---
 
-To point at a Hugging Face Inference Endpoint or any OpenAI-compatible API:
+## 🤖 Inference Pipeline
 
-```bash
-docker run --rm \
-  -e HF_TOKEN=your_hf_token_here \
-  -e API_BASE_URL=https://api-inference.huggingface.co/v1 \
-  -e MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct \
-  ai-code-review-env:latest
 ```
-
-### Run Locally (without Docker)
-
-```bash
-pip install -r requirements.txt
-
-export HF_TOKEN=your_hf_token_here
-export API_BASE_URL=https://api.openai.com/v1
-export MODEL_NAME=gpt-4.1-mini
-
-python inference.py
+1. Read HF_TOKEN, API_BASE_URL, MODEL_NAME
+2. Initialise OpenAI client + environment
+3. Reset environment
+4. Identify → Fix for each task
+5. Log results in required format
 ```
 
 ---
 
-## 🤖 How `inference.py` Works
-
-`inference.py` is the hackathon-required entry point. It wires the LLM agent
-to the environment and runs a full episode.
-
-**Flow:**
+## 🧾 Log Format (Strict)
 
 ```
-1. Read env vars (HF_TOKEN, API_BASE_URL, MODEL_NAME)
-2. Initialise OpenAI client + CodeReviewEnvironment
-3. Print [START] log line
-4. env.reset() → get first observation
-5. Loop:
-   a. Build identify prompt from observation
-   b. Call LLM → get explanation
-   c. env.step(Action("identify", explanation)) → log [STEP]
-   d. Build fix prompt (includes prior identify response for context)
-   e. Call LLM → get corrected code
-   f. env.step(Action("fix", corrected_code)) → log [STEP]
-   g. If done → break
-6. Print [END] log line
+[START]
+[STEP]
+[STEP]
+...
+[END]
 ```
 
-**Design choices:**
-
-- `temperature=0.0` for deterministic LLM output
-- The fix prompt includes the agent's own identify response as context,
-  encouraging coherent two-step reasoning
-- All LLM call failures are caught per-step — the episode continues and
-  the error is recorded in the `[STEP]` log without crashing
-- `[END]` is always printed, even if an unrecoverable error occurs
-
 ---
 
-## 📊 Baseline Results
-
-Below is example output from a full episode using `gpt-4.1-mini`:
+## 📊 Example Output
 
 ```
 [START] task=code-review env=ai-code-review-env model=gpt-4.1-mini
@@ -371,23 +234,24 @@ Below is example output from a full episode using `gpt-4.1-mini`:
 [END] success=true steps=6 rewards=0.71,1.00,0.60,1.00,0.45,1.00
 ```
 
-### Score Breakdown
+---
 
-| Task | Difficulty  | Identify | Fix  | Notes                                        |
-|------|-------------|----------|------|----------------------------------------------|
-| 1    | Easy        | 0.71     | 1.00 | Correct fix + identify bonus (clamped to 1.0)|
-| 2    | Medium      | 0.60     | 1.00 | Correct fix + identify bonus (clamped to 1.0)|
-| 3    | Hard        | 0.45     | 1.00 | Correct fix + identify bonus (clamped to 1.0)|
+## 📌 Why This Stands Out
 
-**Total reward: 4.76 / ~6.0 max**
+* Real-world problem (not synthetic)
+* Deterministic evaluation (fair + reproducible)
+* Reward shaping (better learning signal)
+* Two-phase reasoning (prevents shortcutting)
 
-The model consistently identifies the correct fix across all three tasks.
-Identify scores decrease with difficulty — the performance task has the most
-keywords to cover (`performance`, `optimize`, `forEach`, `iteration`, etc.),
-so partial credit is lower even when the agent's explanation is qualitatively correct.
+---
 
-> **The environment is designed to run within strict resource constraints (2 vCPU,
-> 8GB RAM), ensuring fast evaluation cycles and scalability across many agent runs.**
+## ⚙️ Setup
+
+```bash
+pip install -r requirements.txt
+export HF_TOKEN=your_token
+python inference.py
+```
 
 ---
 
@@ -395,21 +259,28 @@ so partial credit is lower even when the agent's explanation is qualitatively co
 
 ```
 code-review-env/
-├── inference.py          # Entry point (hackathon required, at root)
-├── openenv.yaml          # Environment manifest
-├── Dockerfile            # Container definition
-├── requirements.txt      # Minimal dependencies
-├── README.md             # This file
+├── inference.py
+├── openenv.yaml
+├── Dockerfile
+├── README.md
 └── env/
-    ├── __init__.py
-    ├── models.py         # Observation, Action, EpisodeState dataclasses
-    ├── tasks.py          # 3 task definitions (easy → medium → hard)
-    ├── grader.py         # Deterministic identify + fix graders
-    └── environment.py    # Core environment: reset(), step(), state
 ```
 
 ---
 
+## 🏆 Evaluation Alignment
+
+| Criteria           | Covered |
+| ------------------ | ------- |
+| Real-world utility | ✅       |
+| Task design        | ✅       |
+| Grader quality     | ✅       |
+| Environment design | ✅       |
+| Spec compliance    | ✅       |
+
+---
+
 ## 📄 License
 
-MIT License. Built for the Meta × Hugging Face × OpenEnv × Scaler hackathon.
+MIT License
+Built for Meta × Hugging Face × OpenEnv × Scaler Hackathon 2026