initial submission — Dev Environment Debugger OpenEnv hackathon

Dockerfile

@@ -0,0 +1,19 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy source
+COPY . .
+
+# HuggingFace Spaces uses port 7860
+EXPOSE 7860
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
+  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:7860/health')"
+
+CMD ["python", "api.py"]

README.md

@@ -0,0 +1,242 @@
+# Dev Environment Debugger
+
+An [OpenEnv](https://huggingface.co/openenv) environment where an AI agent debugs a broken multi-service development stack. The agent must read logs, inspect configurations, fix environment variables, and restart services to restore a healthy system — exactly as a senior engineer would during onboarding or an incident.
+
+---
+
+## Motivation
+
+Setting up and debugging a local development environment is one of the most time-consuming tasks in software engineering. Developers routinely lose days diagnosing misconfigured ports, missing env vars, and cascading service failures. No existing agent benchmark targets this problem.
+
+This environment fills that gap: it tests whether an agent can **reason across multiple failure signals, take stateful corrective actions, and know when the system is healthy**.
+
+---
+
+## Environment Description
+
+Simulates a realistic 4-service application stack:
+
+```
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+│   Reverse    │────▶│   Backend    │────▶│   Database   │
+│    Proxy     │     │     API      │     │  (Postgres)  │
+└──────────────┘     └──────┬───────┘     └──────────────┘
+                            │
+                     ┌──────▼───────┐
+                     │    Worker    │
+                     │   Process    │
+                     └──────────────┘
+```
+
+All services run **in memory** — no real Docker, no real ports. The environment is fully deterministic and self-contained.
+
+At the start of each episode, one or more faults are injected (wrong env vars, missing config). The agent must fix them all.
+
+---
+
+## Observation Space
+
+```json
+{
+  "step": 3,
+  "services": {
+    "proxy":    { "status": "healthy", "port": 80   },
+    "api":      { "status": "error",   "port": 8000 },
+    "database": { "status": "healthy", "port": 5432 },
+    "worker":   { "status": "error",   "port": null }
+  },
+  "last_action_result": {
+    "success": false,
+    "message": "API failed to start: cannot connect to database on port 5433"
+  },
+  "available_actions": ["read_logs", "inspect_env", "edit_env", "restart_service", "run_healthcheck", "submit"],
+  "done": false
+}
+```
+
+---
+
+## Action Space
+
+| Action | Required Params | Description |
+|---|---|---|
+| `read_logs` | `service` | Read error logs for a service |
+| `inspect_env` | `service` | View environment variables for a service |
+| `edit_env` | `service`, `key`, `value` | Change an environment variable |
+| `restart_service` | `service` | Restart a service after config fix |
+| `run_healthcheck` | `service` | Get current health status |
+| `submit` | — | Declare environment fixed, end episode |
+
+Valid services: `api`, `worker`, `database`, `proxy`
+
+---
+
+## Reward Function
+
+| Event | Reward |
+|---|---|
+| Read logs before fixing (correct diagnosis) | +0.05 |
+| Inspect env vars | +0.02 |
+| Correct env var edit (fixes a known fault) | +0.10 |
+| Service transitions unhealthy → healthy | +0.30 |
+| Submit with all services healthy | +0.50 |
+| Restart service without fixing root cause | -0.10 |
+| Redundant/invalid action | -0.05 |
+| Submit while services still unhealthy | -0.30 |
+| Exceed max steps (25) | -0.20 |
+
+---
+
+## Tasks
+
+### Task 1 — Single Service Fault (Easy)
+One service is down because `DB_PORT` is set to `5433` instead of `5432`.
+
+**Agent must:** read API logs → inspect API env → fix `DB_PORT` → restart API → submit.
+
+**Expected score:** ~0.85 (GPT-4o-mini)
+
+---
+
+### Task 2 — Multi-Fault Diagnosis (Medium)
+Two independent faults:
+- `DB_PORT` wrong on API service
+- `WORKER_QUEUE_URL` missing on worker service
+
+**Agent must:** identify and fix both faults independently. Partial score for fixing one.
+
+**Expected score:** ~0.60 (GPT-4o-mini)
+
+---
+
+### Task 3 — Cascading Failure Debug (Hard)
+`DB_PASSWORD` is wrong on the database. This causes:
+- Database → auth failure (`FATAL: password authentication failed`)
+- API → misleading `connection timeout` error
+- Worker and Proxy → downstream failures
+
+**Agent must:** trace root cause (database credentials) rather than patching symptoms. Misleading logs make this hard.
+
+**Expected score:** ~0.35 (GPT-4o-mini)
+
+---
+
+## API Endpoints
+
+| Method | Endpoint | Description |
+|---|---|---|
+| POST | `/reset?task_id=task1` | Start new episode |
+| POST | `/step` | Submit action, get observation + reward |
+| GET | `/state` | Current environment state |
+| GET | `/tasks` | Task list and action schema |
+| GET | `/grader?task_id=task1` | Score current episode |
+| POST | `/baseline` | Run baseline agent on all tasks |
+| GET | `/health` | Health check |
+
+---
+
+## Setup & Running
+
+### Local (Python)
+
+```bash
+git clone https://github.com/your-username/dev-env-debugger
+cd dev-env-debugger
+pip install -r requirements.txt
+python api.py
+# Server starts at http://localhost:7860
+```
+
+### Docker
+
+```bash
+docker build -t dev-env-debugger .
+docker run -p 7860:7860 dev-env-debugger
+```
+
+### Validate OpenEnv spec
+
+```bash
+openenv validate
+```
+
+---
+
+## Running the Baseline
+
+```bash
+export OPENAI_API_KEY=your_key_here
+
+# Run all tasks
+python baseline.py
+
+# Run single task
+python baseline.py --task task1
+
+# JSON output (used by /baseline endpoint)
+python baseline.py --output-json
+```
+
+### Baseline Scores (GPT-4o-mini)
+
+| Task | Difficulty | Expected Score |
+|---|---|---|
+| task1 | Easy | ~0.85 |
+| task2 | Medium | ~0.60 |
+| task3 | Hard | ~0.35 |
+| **Average** | | **~0.60** |
+
+---
+
+## Quick API Example
+
+```bash
+# Start episode
+curl -X POST "http://localhost:7860/reset?task_id=task1"
+
+# Read logs
+curl -X POST http://localhost:7860/step \
+  -H "Content-Type: application/json" \
+  -d '{"type": "read_logs", "service": "api"}'
+
+# Fix env var
+curl -X POST http://localhost:7860/step \
+  -H "Content-Type: application/json" \
+  -d '{"type": "edit_env", "service": "api", "key": "DB_PORT", "value": "5432"}'
+
+# Restart service
+curl -X POST http://localhost:7860/step \
+  -H "Content-Type: application/json" \
+  -d '{"type": "restart_service", "service": "api"}'
+
+# Submit
+curl -X POST http://localhost:7860/step \
+  -H "Content-Type: application/json" \
+  -d '{"type": "submit"}'
+
+# Get score
+curl "http://localhost:7860/grader?task_id=task1"
+```
+
+---
+
+## Project Structure
+
+```
+dev-env-debugger/
+├── environment.py   # Core simulation — state machine, fault injection, action handlers
+├── tasks.py         # Task definitions and programmatic graders
+├── models.py        # Pydantic models (Observation, Action, Reward, ServiceStatus)
+├── api.py           # FastAPI server exposing all endpoints
+├── baseline.py      # LLM baseline inference script
+├── openenv.yaml     # OpenEnv metadata
+├── requirements.txt
+├── Dockerfile
+└── README.md
+```
+
+---
+
+## License
+
+MIT

api.py

@@ -0,0 +1,151 @@
+"""
+FastAPI server — exposes OpenEnv standard endpoints + required hackathon endpoints.
+"""
+
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import HTMLResponse
+import uvicorn
+
+from models import Action, Observation
+from environment import DevEnvEnvironment
+from tasks import TASKS, grade_episode
+
+app = FastAPI(
+    title="Dev Environment Debugger",
+    description="OpenEnv environment where an AI agent debugs a broken multi-service dev stack.",
+    version="1.0.0",
+)
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# One environment instance per server (single-session for hackathon scope)
+env = DevEnvEnvironment()
+
+
+# ---------------------------------------------------------------------------
+# OpenEnv standard endpoints
+# ---------------------------------------------------------------------------
+
+@app.post("/reset")
+def reset(task_id: str = "task1") -> dict:
+    """Start a new episode. Returns initial observation."""
+    if task_id not in TASKS:
+        raise HTTPException(status_code=400, detail=f"Unknown task_id. Valid: {list(TASKS.keys())}")
+    obs = env.reset(task_id=task_id)
+    return obs.dict()
+
+
+@app.post("/step")
+def step(action: Action) -> dict:
+    """Submit an action. Returns observation, reward, done, info."""
+    obs, reward, done, info = env.step(action)
+    return {
+        "observation": obs.dict(),
+        "reward": reward,
+        "done": done,
+        "info": info,
+    }
+
+
+@app.get("/state")
+def state() -> dict:
+    """Returns current environment state."""
+    return env.state()
+
+
+# ---------------------------------------------------------------------------
+# Required hackathon endpoints
+# ---------------------------------------------------------------------------
+
+@app.get("/tasks")
+def list_tasks() -> dict:
+    """Returns task list and action schema."""
+    action_schema = {
+        "type": "string — one of: read_logs | inspect_env | edit_env | restart_service | run_healthcheck | submit",
+        "service": "string — target service: api | worker | database | proxy",
+        "key": "string — env var key (only for edit_env)",
+        "value": "string — env var value (only for edit_env)",
+    }
+    return {
+        "tasks": [
+            {
+                "id": t["id"],
+                "name": t["name"],
+                "difficulty": t["difficulty"],
+                "description": t["description"],
+                "faults_count": len(t["faults"]),
+            }
+            for t in TASKS.values()
+        ],
+        "action_schema": action_schema,
+    }
+
+
+@app.get("/grader")
+def grader(task_id: str = None) -> dict:
+    """Returns grader score for current or specified episode."""
+    current_state = env.state()
+    tid = task_id or current_state.get("task_id", "task1")
+    result = grade_episode(tid, current_state)
+    return {
+        "task_id": tid,
+        "grader_result": result,
+        "state_summary": {
+            "step": current_state["step"],
+            "fixed_faults": current_state["fixed_faults"],
+            "remaining_faults": current_state["remaining_faults"],
+            "services": {k: v["status"] for k, v in current_state["services"].items()},
+        },
+    }
+
+
+@app.post("/baseline")
+def baseline() -> dict:
+    """
+    Runs the baseline inference script against all 3 tasks.
+    Returns scores. Requires OPENAI_API_KEY in environment.
+    """
+    try:
+        import subprocess
+        import json
+        result = subprocess.run(
+            ["python", "baseline.py", "--output-json"],
+            capture_output=True, text=True, timeout=300
+        )
+        if result.returncode == 0:
+            scores = json.loads(result.stdout)
+            return {"status": "success", "scores": scores}
+        else:
+            return {"status": "error", "message": result.stderr}
+    except Exception as e:
+        return {"status": "error", "message": str(e)}
+
+
+# ---------------------------------------------------------------------------
+# Health check
+# ---------------------------------------------------------------------------
+
+@app.get("/")
+def root():
+    return {
+        "name": "Dev Environment Debugger",
+        "version": "1.0.0",
+        "description": "OpenEnv environment for AI-powered dev environment debugging.",
+        "endpoints": ["/reset", "/step", "/state", "/tasks", "/grader", "/baseline"],
+        "openenv": True,
+    }
+
+
+@app.get("/health")
+def health():
+    return {"status": "ok"}
+
+
+if __name__ == "__main__":
+    uvicorn.run("api:app", host="0.0.0.0", port=7860, reload=False)

baseline.py

@@ -0,0 +1,214 @@
+"""
+Baseline inference script.
+Runs an LLM agent (via OpenAI API) against all 3 tasks and reports scores.
+
+Usage:
+    export OPENAI_API_KEY=your_key_here
+    python baseline.py
+
+    # For JSON output (used by /baseline endpoint):
+    python baseline.py --output-json
+"""
+
+import os
+import sys
+import json
+import argparse
+import requests
+
+# ---------------------------------------------------------------------------
+# Config
+# ---------------------------------------------------------------------------
+
+API_BASE = os.getenv("ENV_BASE_URL", "http://localhost:7860")
+OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "")
+MODEL = os.getenv("BASELINE_MODEL", "gpt-4o-mini")
+MAX_STEPS = 20
+
+SYSTEM_PROMPT = """You are an expert DevOps engineer debugging a broken multi-service development environment.
+
+You will receive the current state of services (healthy/error) and the result of your last action.
+Your goal is to diagnose faults and fix them so ALL services reach "healthy" status.
+
+Services: api, worker, database, proxy
+Available actions:
+  - read_logs: read error logs for a service (do this first when a service is broken)
+  - inspect_env: view environment variables for a service
+  - edit_env: change an environment variable (requires key and value)
+  - restart_service: restart a service after fixing its config
+  - run_healthcheck: check current status of a service
+  - submit: declare done when all services are healthy
+
+STRATEGY:
+1. Look at which services are in "error" state
+2. Read logs for broken services to understand the error
+3. Inspect env vars of the broken service
+4. Edit the wrong/missing env var
+5. Restart the service
+6. Submit when all services show "healthy"
+
+Respond ONLY with a valid JSON action object. Nothing else. Examples:
+{"type": "read_logs", "service": "api"}
+{"type": "inspect_env", "service": "database"}  
+{"type": "edit_env", "service": "api", "key": "DB_PORT", "value": "5432"}
+{"type": "restart_service", "service": "api"}
+{"type": "submit"}
+"""
+
+
+def call_openai(messages: list) -> str:
+    """Call OpenAI chat completions API."""
+    if not OPENAI_API_KEY:
+        raise ValueError("OPENAI_API_KEY not set in environment variables.")
+
+    import urllib.request
+    payload = {
+        "model": MODEL,
+        "messages": messages,
+        "temperature": 0.2,
+        "max_tokens": 200,
+    }
+    req = urllib.request.Request(
+        "https://api.openai.com/v1/chat/completions",
+        data=json.dumps(payload).encode(),
+        headers={
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {OPENAI_API_KEY}",
+        },
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=30) as resp:
+        data = json.loads(resp.read())
+    return data["choices"][0]["message"]["content"].strip()
+
+
+def obs_to_text(obs: dict) -> str:
+    """Convert observation dict to readable text for the LLM."""
+    services = obs.get("services", {})
+    svc_lines = "\n".join(
+        f"  {name}: {s['status']} (port: {s.get('port', 'N/A')})"
+        for name, s in services.items()
+    )
+    last = obs.get("last_action_result", {})
+    return (
+        f"Step {obs['step']}\n"
+        f"Services:\n{svc_lines}\n"
+        f"Last action result: {'✓' if last.get('success') else '✗'} {last.get('message', '')}\n"
+        f"Done: {obs.get('done', False)}"
+    )
+
+
+def run_episode(task_id: str, verbose: bool = True) -> dict:
+    """Run one full episode for a task. Returns grader result."""
+    if verbose:
+        print(f"\n{'='*60}")
+        print(f"Running Task: {task_id}")
+        print('='*60)
+
+    # Reset
+    resp = requests.post(f"{API_BASE}/reset", params={"task_id": task_id})
+    obs = resp.json()
+
+    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
+    step = 0
+
+    while not obs.get("done", False) and step < MAX_STEPS:
+        step += 1
+        obs_text = obs_to_text(obs)
+
+        if verbose:
+            print(f"\n[Step {step}] Observation:\n{obs_text}")
+
+        messages.append({"role": "user", "content": obs_text})
+
+        # Get action from LLM
+        try:
+            raw_action = call_openai(messages)
+        except Exception as e:
+            if verbose:
+                print(f"  LLM error: {e}")
+            break
+
+        if verbose:
+            print(f"  Agent action: {raw_action}")
+
+        # Parse and validate action
+        try:
+            action_dict = json.loads(raw_action)
+        except json.JSONDecodeError:
+            if verbose:
+                print(f"  Could not parse action JSON: {raw_action}")
+            messages.append({"role": "assistant", "content": raw_action})
+            messages.append({"role": "user", "content": "Invalid JSON. Respond ONLY with a valid JSON action."})
+            continue
+
+        messages.append({"role": "assistant", "content": raw_action})
+
+        # Step environment
+        step_resp = requests.post(f"{API_BASE}/step", json=action_dict)
+        step_data = step_resp.json()
+        obs = step_data["observation"]
+        reward = step_data["reward"]
+        done = step_data["done"]
+
+        if verbose:
+            print(f"  Reward: {reward} | Done: {done}")
+
+        if done:
+            break
+
+    # Get grader score
+    grader_resp = requests.get(f"{API_BASE}/grader", params={"task_id": task_id})
+    grader_data = grader_resp.json()
+    result = grader_data["grader_result"]
+
+    if verbose:
+        print(f"\n[RESULT] Task {task_id}: score={result['score']} | {result['reason']}")
+
+    return {
+        "task_id": task_id,
+        "score": result["score"],
+        "reason": result["reason"],
+        "steps_taken": step,
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--output-json", action="store_true", help="Output scores as JSON")
+    parser.add_argument("--task", type=str, default=None, help="Run single task only")
+    args = parser.parse_args()
+
+    verbose = not args.output_json
+    tasks = [args.task] if args.task else ["task1", "task2", "task3"]
+
+    results = []
+    for task_id in tasks:
+        try:
+            result = run_episode(task_id, verbose=verbose)
+            results.append(result)
+        except Exception as e:
+            results.append({
+                "task_id": task_id,
+                "score": 0.0,
+                "reason": f"Error: {str(e)}",
+                "steps_taken": 0,
+            })
+
+    avg_score = sum(r["score"] for r in results) / len(results)
+
+    if args.output_json:
+        output = {"tasks": results, "average_score": round(avg_score, 3)}
+        print(json.dumps(output))
+    else:
+        print(f"\n{'='*60}")
+        print("BASELINE SCORES SUMMARY")
+        print('='*60)
+        for r in results:
+            print(f"  {r['task_id']:10} score={r['score']:.3f}  steps={r['steps_taken']}  {r['reason']}")
+        print(f"  {'AVERAGE':10} score={avg_score:.3f}")
+        print('='*60)
+
+
+if __name__ == "__main__":
+    main()

environment.py

@@ -0,0 +1,391 @@
+"""
+Dev Environment Debugger - Core Environment
+Simulates a broken multi-service dev stack in memory.
+Agent must diagnose and fix faults to bring all services healthy.
+"""
+
+import copy
+import random
+from typing import Any
+from models import Observation, Action, Reward, ServiceStatus, ActionResult
+
+# ---------------------------------------------------------------------------
+# Fault definitions
+# ---------------------------------------------------------------------------
+
+FAULT_CATALOG = {
+    "wrong_db_port": {
+        "service": "api",
+        "description": "DB_PORT set to wrong value",
+        "env_key": "DB_PORT",
+        "bad_value": "5433",
+        "fix_key": "DB_PORT",
+        "fix_value": "5432",
+        "log": "Error: cannot connect to database on port 5433. Connection refused.",
+        "requires_restart": ["api", "worker"],
+    },
+    "missing_queue_url": {
+        "service": "worker",
+        "description": "WORKER_QUEUE_URL not set",
+        "env_key": "WORKER_QUEUE_URL",
+        "bad_value": None,  # missing
+        "fix_key": "WORKER_QUEUE_URL",
+        "fix_value": "redis://localhost:6379",
+        "log": "Error: WORKER_QUEUE_URL is not set. Worker cannot connect to queue.",
+        "requires_restart": ["worker"],
+    },
+    "wrong_api_port": {
+        "service": "proxy",
+        "description": "PROXY_UPSTREAM points to wrong API port",
+        "env_key": "PROXY_UPSTREAM",
+        "bad_value": "http://api:9999",
+        "fix_key": "PROXY_UPSTREAM",
+        "fix_value": "http://api:8000",
+        "log": "Error: upstream http://api:9999 unreachable. 502 Bad Gateway.",
+        "requires_restart": ["proxy"],
+    },
+    "wrong_db_password": {
+        "service": "database",
+        "description": "DB_PASSWORD is incorrect causing auth failure",
+        "env_key": "DB_PASSWORD",
+        "bad_value": "wrongpassword",
+        "fix_key": "DB_PASSWORD",
+        "fix_value": "secret",
+        "log": "FATAL: password authentication failed for user 'app'. Check DB_PASSWORD.",
+        "requires_restart": ["database", "api", "worker"],
+    },
+}
+
+# ---------------------------------------------------------------------------
+# Service dependency graph — which services depend on which
+# ---------------------------------------------------------------------------
+
+DEPENDENCIES = {
+    "proxy":    ["api"],
+    "api":      ["database"],
+    "worker":   ["database"],
+    "database": [],
+}
+
+# Default healthy env vars per service
+DEFAULT_ENV = {
+    "api": {
+        "DB_HOST": "localhost",
+        "DB_PORT": "5432",
+        "DB_PASSWORD": "secret",
+        "API_PORT": "8000",
+    },
+    "worker": {
+        "WORKER_QUEUE_URL": "redis://localhost:6379",
+        "API_HOST": "http://api:8000",
+    },
+    "database": {
+        "DB_PASSWORD": "secret",
+        "DB_PORT": "5432",
+    },
+    "proxy": {
+        "PROXY_UPSTREAM": "http://api:8000",
+        "PROXY_PORT": "80",
+    },
+}
+
+SERVICE_PORTS = {
+    "api": 8000,
+    "worker": None,
+    "database": 5432,
+    "proxy": 80,
+}
+
+MAX_STEPS = 25
+
+
+# ---------------------------------------------------------------------------
+# Environment class
+# ---------------------------------------------------------------------------
+
+class DevEnvEnvironment:
+    def __init__(self):
+        self._state: dict[str, Any] = {}
+        self._active_faults: list[str] = []
+        self._fixed_faults: set[str] = set()
+        self._step_count: int = 0
+        self._done: bool = False
+        self._task_id: str = "task1"
+        self._log_history: list[str] = []
+
+    # ------------------------------------------------------------------
+    # OpenEnv API
+    # ------------------------------------------------------------------
+
+    def reset(self, task_id: str = "task1") -> Observation:
+        self._task_id = task_id
+        self._step_count = 0
+        self._done = False
+        self._fixed_faults = set()
+        self._log_history = []
+
+        # Load task faults
+        from tasks import TASKS
+        task = TASKS[task_id]
+        self._active_faults = list(task["faults"])
+
+        # Build initial service env state from defaults
+        self._state = {
+            "envs": copy.deepcopy(DEFAULT_ENV),
+            "services": {},
+        }
+
+        # Inject faults
+        for fault_id in self._active_faults:
+            fault = FAULT_CATALOG[fault_id]
+            svc = fault["service"]
+            key = fault["env_key"]
+            if fault["bad_value"] is None:
+                # Remove the key (missing env var)
+                self._state["envs"][svc].pop(key, None)
+            else:
+                self._state["envs"][svc][key] = fault["bad_value"]
+
+        # Compute initial service statuses
+        self._recompute_statuses()
+
+        return self._build_observation(
+            last_action_result=ActionResult(success=True, message="Environment initialized. Services started.")
+        )
+
+    def step(self, action: Action) -> tuple[Observation, float, bool, dict]:
+        if self._done:
+            obs = self._build_observation(
+                ActionResult(success=False, message="Episode already done. Call reset().")
+            )
+            return obs, 0.0, True, {}
+
+        self._step_count += 1
+        reward = 0.0
+        result: ActionResult
+
+        # Dispatch action
+        if action.type == "read_logs":
+            result, r = self._handle_read_logs(action)
+            reward += r
+        elif action.type == "inspect_env":
+            result, r = self._handle_inspect_env(action)
+            reward += r
+        elif action.type == "edit_env":
+            result, r = self._handle_edit_env(action)
+            reward += r
+        elif action.type == "restart_service":
+            result, r = self._handle_restart_service(action)
+            reward += r
+        elif action.type == "run_healthcheck":
+            result, r = self._handle_healthcheck(action)
+            reward += r
+        elif action.type == "submit":
+            result, r, self._done = self._handle_submit()
+            reward += r
+        else:
+            result = ActionResult(success=False, message=f"Unknown action type: {action.type}")
+            reward -= 0.05
+
+        # Penalty for exceeding steps
+        if self._step_count >= MAX_STEPS and not self._done:
+            self._done = True
+            reward -= 0.2
+            result.message += " | Max steps exceeded."
+
+        self._log_history.append(f"Step {self._step_count}: {action.type} → {result.message}")
+
+        obs = self._build_observation(result)
+        return obs, round(reward, 3), self._done, {"step": self._step_count}
+
+    def state(self) -> dict:
+        return {
+            "task_id": self._task_id,
+            "step": self._step_count,
+            "done": self._done,
+            "active_faults": self._active_faults,
+            "fixed_faults": list(self._fixed_faults),
+            "remaining_faults": [f for f in self._active_faults if f not in self._fixed_faults],
+            "services": {k: v.dict() for k, v in self._state["services"].items()},
+            "envs": self._state["envs"],
+        }
+
+    # ------------------------------------------------------------------
+    # Action handlers
+    # ------------------------------------------------------------------
+
+    def _handle_read_logs(self, action: Action) -> tuple[ActionResult, float]:
+        svc = action.service
+        if svc not in self._state["services"]:
+            return ActionResult(success=False, message=f"Unknown service: {svc}"), -0.05
+
+        status = self._state["services"][svc]
+        reward = 0.0
+
+        if status.status == "error":
+            # Find fault log for this service
+            logs = []
+            for fault_id in self._active_faults:
+                if fault_id not in self._fixed_faults:
+                    fault = FAULT_CATALOG[fault_id]
+                    if fault["service"] == svc:
+                        logs.append(fault["log"])
+            # Also add downstream symptom logs
+            for fault_id in self._active_faults:
+                if fault_id not in self._fixed_faults:
+                    fault = FAULT_CATALOG[fault_id]
+                    if svc in fault.get("requires_restart", []) and fault["service"] != svc:
+                        logs.append(f"[downstream] Service unavailable due to dependency failure.")
+
+            log_text = "\n".join(logs) if logs else "No errors in log."
+            reward += 0.05  # reward for reading logs before acting
+            return ActionResult(success=True, message=f"[{svc} logs]\n{log_text}"), reward
+        else:
+            return ActionResult(success=True, message=f"[{svc} logs]\nNo errors. Service running normally."), 0.0
+
+    def _handle_inspect_env(self, action: Action) -> tuple[ActionResult, float]:
+        svc = action.service
+        if svc not in self._state["envs"]:
+            return ActionResult(success=False, message=f"Unknown service: {svc}"), -0.05
+
+        env = self._state["envs"][svc]
+        env_text = "\n".join(f"  {k}={v}" for k, v in env.items())
+        return ActionResult(success=True, message=f"[{svc} env]\n{env_text}"), 0.02
+
+    def _handle_edit_env(self, action: Action) -> tuple[ActionResult, float]:
+        svc = action.service
+        if not action.key or action.value is None:
+            return ActionResult(success=False, message="edit_env requires 'key' and 'value' params."), -0.05
+        if svc not in self._state["envs"]:
+            return ActionResult(success=False, message=f"Unknown service: {svc}"), -0.05
+
+        self._state["envs"][svc][action.key] = action.value
+
+        # Check if this edit fixes a fault
+        reward = 0.0
+        for fault_id in self._active_faults:
+            if fault_id in self._fixed_faults:
+                continue
+            fault = FAULT_CATALOG[fault_id]
+            if (fault["service"] == svc and
+                    fault["fix_key"] == action.key and
+                    fault["fix_value"] == action.value):
+                # Correct fix applied — mark as pending restart
+                reward += 0.1
+                return ActionResult(
+                    success=True,
+                    message=f"Updated {svc}.{action.key}={action.value}. Restart {svc} to apply."
+                ), reward
+
+        return ActionResult(
+            success=True,
+            message=f"Updated {svc}.{action.key}={action.value}."
+        ), reward
+
+    def _handle_restart_service(self, action: Action) -> tuple[ActionResult, float]:
+        svc = action.service
+        if svc not in self._state["services"]:
+            return ActionResult(success=False, message=f"Unknown service: {svc}"), -0.05
+
+        reward = 0.0
+        prev_statuses = {s: self._state["services"][s].status for s in self._state["services"]}
+
+        # Check if fault is now fixed (env has correct value)
+        for fault_id in self._active_faults:
+            if fault_id in self._fixed_faults:
+                continue
+            fault = FAULT_CATALOG[fault_id]
+            if svc in fault["requires_restart"]:
+                current_val = self._state["envs"].get(fault["service"], {}).get(fault["fix_key"])
+                if current_val == fault["fix_value"]:
+                    self._fixed_faults.add(fault_id)
+
+        self._recompute_statuses()
+
+        # Check improvements
+        for s, status_obj in self._state["services"].items():
+            if prev_statuses[s] != "healthy" and status_obj.status == "healthy":
+                reward += 0.3  # service recovered
+
+        # Penalty: restarted without fixing root cause
+        if self._state["services"][svc].status == "error":
+            reward -= 0.1
+            return ActionResult(
+                success=True,
+                message=f"{svc} restarted but still in error state. Root cause not fixed."
+            ), reward
+
+        return ActionResult(success=True, message=f"{svc} restarted successfully."), reward
+
+    def _handle_healthcheck(self, action: Action) -> tuple[ActionResult, float]:
+        svc = action.service
+        if svc not in self._state["services"]:
+            return ActionResult(success=False, message=f"Unknown service: {svc}"), -0.05
+
+        status = self._state["services"][svc]
+        return ActionResult(
+            success=True,
+            message=f"[{svc} healthcheck] status={status.status} port={status.port}"
+        ), 0.01
+
+    def _handle_submit(self) -> tuple[ActionResult, float, bool]:
+        all_healthy = all(
+            s.status == "healthy" for s in self._state["services"].values()
+        )
+        if all_healthy:
+            return ActionResult(
+                success=True,
+                message="All services healthy. Environment fixed successfully!"
+            ), 0.5, True
+        else:
+            unhealthy = [k for k, v in self._state["services"].items() if v.status != "healthy"]
+            return ActionResult(
+                success=False,
+                message=f"Submit failed. Services still unhealthy: {unhealthy}"
+            ), -0.3, True
+
+    # ------------------------------------------------------------------
+    # Internal helpers
+    # ------------------------------------------------------------------
+
+    def _recompute_statuses(self):
+        """Recompute service statuses based on active unfixed faults and dependencies."""
+        # Start all healthy
+        statuses = {svc: "healthy" for svc in DEFAULT_ENV}
+
+        # Apply unfixed faults
+        remaining = [f for f in self._active_faults if f not in self._fixed_faults]
+        directly_broken = set()
+        for fault_id in remaining:
+            fault = FAULT_CATALOG[fault_id]
+            directly_broken.add(fault["service"])
+
+        # Propagate through dependency graph
+        for svc in ["database", "api", "worker", "proxy"]:
+            if svc in directly_broken:
+                statuses[svc] = "error"
+            else:
+                # Check if any dependency is broken
+                for dep in DEPENDENCIES.get(svc, []):
+                    if statuses.get(dep) == "error":
+                        statuses[svc] = "error"
+                        break
+
+        self._state["services"] = {
+            svc: ServiceStatus(
+                name=svc,
+                status=statuses[svc],
+                port=SERVICE_PORTS[svc]
+            )
+            for svc in DEFAULT_ENV
+        }
+
+    def _build_observation(self, last_action_result: ActionResult) -> Observation:
+        return Observation(
+            step=self._step_count,
+            services=self._state["services"],
+            last_action_result=last_action_result,
+            available_actions=["read_logs", "inspect_env", "edit_env",
+                               "restart_service", "run_healthcheck", "submit"],
+            done=self._done,
+        )

models.py

@@ -0,0 +1,45 @@
+"""
+Typed Pydantic models for OpenEnv spec compliance.
+"""
+
+from typing import Optional
+from pydantic import BaseModel
+
+
+class ServiceStatus(BaseModel):
+    name: str
+    status: str  # "healthy" | "error" | "stopped"
+    port: Optional[int] = None
+
+
+class ActionResult(BaseModel):
+    success: bool
+    message: str
+
+
+class Observation(BaseModel):
+    step: int
+    services: dict[str, ServiceStatus]
+    last_action_result: ActionResult
+    available_actions: list[str]
+    done: bool
+
+
+class Action(BaseModel):
+    type: str                        # read_logs | inspect_env | edit_env | restart_service | run_healthcheck | submit
+    service: Optional[str] = None   # target service name
+    key: Optional[str] = None       # for edit_env
+    value: Optional[str] = None     # for edit_env
+
+
+class Reward(BaseModel):
+    value: float
+    reason: str
+
+
+class TaskInfo(BaseModel):
+    id: str
+    name: str
+    description: str
+    difficulty: str
+    faults: list[str]

openenv.yaml

@@ -0,0 +1,72 @@
+name: dev-env-debugger
+version: 1.0.0
+description: >
+  An OpenEnv environment where an AI agent debugs a broken multi-service
+  development stack. The agent must read logs, inspect configs, fix
+  environment variables, and restart services to restore a healthy state.
+
+tags:
+  - openenv
+  - devops
+  - debugging
+  - real-world
+  - multi-service
+
+author: Ajinkya
+license: MIT
+
+environment:
+  reset_endpoint: /reset
+  step_endpoint: /step
+  state_endpoint: /state
+  tasks_endpoint: /tasks
+  grader_endpoint: /grader
+  baseline_endpoint: /baseline
+
+tasks:
+  - id: task1
+    name: Single Service Fault
+    difficulty: easy
+    description: One service down due to a single misconfigured environment variable.
+
+  - id: task2
+    name: Multi-Fault Diagnosis
+    difficulty: medium
+    description: Two services failing due to independent faults. Partial credit per fault fixed.
+
+  - id: task3
+    name: Cascading Failure Debug
+    difficulty: hard
+    description: Root cause fault causing cascading failures with misleading error messages.
+
+observation_space:
+  type: object
+  fields:
+    step: integer
+    services: object  # map of service name → {status, port}
+    last_action_result: object  # {success: bool, message: string}
+    available_actions: array
+    done: boolean
+
+action_space:
+  type: object
+  fields:
+    type: string  # read_logs | inspect_env | edit_env | restart_service | run_healthcheck | submit
+    service: string  # optional — api | worker | database | proxy
+    key: string      # optional — env var key (edit_env only)
+    value: string    # optional — env var value (edit_env only)
+
+reward:
+  min: -0.5
+  max: 1.0
+  shaped: true
+  description: >
+    Partial rewards for correct diagnostic actions and service recovery.
+    Penalties for redundant actions and submitting while unhealthy.
+
+baseline:
+  model: gpt-4o-mini
+  expected_scores:
+    task1: 0.85
+    task2: 0.60
+    task3: 0.35

requirements.txt

@@ -0,0 +1,5 @@
+fastapi==0.111.0
+uvicorn==0.30.1
+pydantic==2.7.1
+requests==2.32.3
+pyyaml==6.0.1

tasks.py

@@ -0,0 +1,150 @@
+"""
+Task definitions and programmatic graders.
+Each task defines faults to inject and a grader that scores 0.0 - 1.0.
+"""
+
+TASKS = {
+    "task1": {
+        "id": "task1",
+        "name": "Single Service Fault",
+        "difficulty": "easy",
+        "description": (
+            "One service is down due to a single misconfigured environment variable. "
+            "The API cannot connect to the database because DB_PORT is set to the wrong value. "
+            "Read the logs, find the bad config, fix it, and restart the service."
+        ),
+        "faults": ["wrong_db_port"],
+    },
+    "task2": {
+        "id": "task2",
+        "name": "Multi-Fault Diagnosis",
+        "difficulty": "medium",
+        "description": (
+            "Two services are failing due to independent faults. "
+            "The API has a wrong database port AND the worker is missing its queue URL. "
+            "Both must be fixed for full score. Partial credit for fixing one."
+        ),
+        "faults": ["wrong_db_port", "missing_queue_url"],
+    },
+    "task3": {
+        "id": "task3",
+        "name": "Cascading Failure Debug",
+        "difficulty": "hard",
+        "description": (
+            "The database has wrong credentials, causing the API to fail with a misleading "
+            "'connection timeout' error, which in turn causes the worker and proxy to fail. "
+            "Logs are misleading — the agent must trace the root cause (DB_PASSWORD) rather "
+            "than patching downstream symptoms."
+        ),
+        "faults": ["wrong_db_password"],
+    },
+}
+
+
+def grade_episode(task_id: str, env_state: dict) -> dict:
+    """
+    Programmatic grader. Returns score 0.0-1.0 and breakdown.
+    Called after episode ends (done=True).
+    """
+    task = TASKS.get(task_id)
+    if not task:
+        return {"score": 0.0, "reason": f"Unknown task: {task_id}"}
+
+    services = env_state.get("services", {})
+    fixed_faults = set(env_state.get("fixed_faults", []))
+    active_faults = set(env_state.get("active_faults", []))
+    remaining = active_faults - fixed_faults
+    steps = env_state.get("step", 0)
+
+    all_healthy = all(s["status"] == "healthy" for s in services.values())
+    total_faults = len(active_faults)
+    faults_fixed = len(fixed_faults)
+
+    # ---------------------------------------------------------------
+    # Task 1 grader — single fault, binary but with partial for partial fix
+    # ---------------------------------------------------------------
+    if task_id == "task1":
+        if all_healthy and faults_fixed == total_faults:
+            # Efficiency bonus for doing it quickly
+            efficiency = max(0.0, 1.0 - (steps - 3) * 0.03)
+            score = min(1.0, 0.85 + efficiency * 0.15)
+            return {
+                "score": round(score, 3),
+                "reason": f"All services healthy. Fixed in {steps} steps.",
+                "faults_fixed": faults_fixed,
+                "total_faults": total_faults,
+            }
+        elif faults_fixed > 0:
+            return {
+                "score": 0.4,
+                "reason": "Fault fixed but service not restarted or submit not called.",
+                "faults_fixed": faults_fixed,
+                "total_faults": total_faults,
+            }
+        else:
+            return {
+                "score": 0.0,
+                "reason": "No faults fixed.",
+                "faults_fixed": 0,
+                "total_faults": total_faults,
+            }
+
+    # ---------------------------------------------------------------
+    # Task 2 grader — partial credit per fault fixed
+    # ---------------------------------------------------------------
+    if task_id == "task2":
+        if all_healthy and faults_fixed == total_faults:
+            efficiency = max(0.0, 1.0 - (steps - 5) * 0.02)
+            score = min(1.0, 0.85 + efficiency * 0.15)
+            return {
+                "score": round(score, 3),
+                "reason": f"All services healthy. Both faults fixed in {steps} steps.",
+                "faults_fixed": faults_fixed,
+                "total_faults": total_faults,
+            }
+        elif faults_fixed == 1:
+            return {
+                "score": 0.45,
+                "reason": f"One of two faults fixed. Partial credit.",
+                "faults_fixed": faults_fixed,
+                "total_faults": total_faults,
+            }
+        else:
+            return {
+                "score": 0.0,
+                "reason": "No faults fixed.",
+                "faults_fixed": 0,
+                "total_faults": total_faults,
+            }
+
+    # ---------------------------------------------------------------
+    # Task 3 grader — penalizes symptom-patching, rewards root cause fix
+    # ---------------------------------------------------------------
+    if task_id == "task3":
+        if all_healthy and faults_fixed == total_faults:
+            # Reward fewer steps heavily (cascading fix is fast if root cause found)
+            efficiency = max(0.0, 1.0 - (steps - 4) * 0.04)
+            score = min(1.0, 0.80 + efficiency * 0.20)
+            return {
+                "score": round(score, 3),
+                "reason": f"Root cause found and fixed. All services recovered in {steps} steps.",
+                "faults_fixed": faults_fixed,
+                "total_faults": total_faults,
+            }
+        elif faults_fixed > 0:
+            # Fixed something but not all healthy — partial
+            return {
+                "score": 0.3,
+                "reason": "Partial fix. Root cause addressed but services not fully recovered.",
+                "faults_fixed": faults_fixed,
+                "total_faults": total_faults,
+            }
+        else:
+            return {
+                "score": 0.0,
+                "reason": "Root cause not identified. All services still broken.",
+                "faults_fixed": 0,
+                "total_faults": total_faults,
+            }
+
+    return {"score": 0.0, "reason": "Unhandled task."}