README.md

metadata

title: Dev Environment Debugger
emoji: 🐛
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: false

Dev Environment Debugger

An OpenEnv environment where an AI agent debugs a broken multi-service development stack. The agent must read logs, inspect configurations, fix environment variables, and restart services to restore a healthy system — exactly as a senior engineer would during onboarding or an incident.

---

Motivation

Setting up and debugging a local development environment is one of the most time-consuming tasks in software engineering. Developers routinely lose days diagnosing misconfigured ports, missing env vars, and cascading service failures. No existing agent benchmark targets this problem.

This environment fills that gap: it tests whether an agent can reason across multiple failure signals, take stateful corrective actions, and know when the system is healthy.

---

Environment Description

Simulates a realistic 4-service application stack:

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│   Reverse    │────▶│   Backend    │────▶│   Database   │
│    Proxy     │     │     API      │     │  (Postgres)  │
└──────────────┘     └──────┬───────┘     └──────────────┘
                            │
                     ┌──────▼───────┐
                     │    Worker    │
                     │   Process    │
                     └──────────────┘

All services run in memory — no real Docker, no real ports. The environment is fully deterministic and self-contained.

At the start of each episode, one or more faults are injected (wrong env vars, missing config). The agent must fix them all.

---

Observation Space

{
  "step": 3,
  "services": {
    "proxy":    { "status": "healthy", "port": 80   },
    "api":      { "status": "error",   "port": 8000 },
    "database": { "status": "healthy", "port": 5432 },
    "worker":   { "status": "error",   "port": null }
  },
  "last_action_result": {
    "success": false,
    "message": "API failed to start: cannot connect to database on port 5433"
  },
  "available_actions": ["read_logs", "inspect_env", "edit_env", "restart_service", "run_healthcheck", "submit"],
  "done": false
}

---

Action Space

Action	Required Params	Description
read_logs	service	Read error logs for a service
inspect_env	service	View environment variables for a service
edit_env	service, key, value	Change an environment variable
restart_service	service	Restart a service after config fix
run_healthcheck	service	Get current health status
submit	—	Declare environment fixed, end episode

Valid services: api, worker, database, proxy

---

Reward Function

Event	Reward
Read logs before fixing (correct diagnosis)	+0.05
Inspect env vars	+0.02
Correct env var edit (fixes a known fault)	+0.10
Service transitions unhealthy → healthy	+0.30
Submit with all services healthy	+0.50
Restart service without fixing root cause	-0.10
Redundant/invalid action	-0.05
Submit while services still unhealthy	-0.30
Exceed max steps (25)	-0.20

---

Tasks

Task 1 — Single Service Fault (Easy)

One service is down because DB_PORT is set to 5433 instead of 5432.

Agent must: read API logs → inspect API env → fix DB_PORT → restart API → submit.

Expected score: ~0.85 (GPT-4o-mini)

---

Task 2 — Multi-Fault Diagnosis (Medium)

Two independent faults:

• DB_PORT wrong on API service
• WORKER_QUEUE_URL missing on worker service

Agent must: identify and fix both faults independently. Partial score for fixing one.

Expected score: ~0.60 (GPT-4o-mini)

---

Task 3 — Cascading Failure Debug (Hard)

DB_PASSWORD is wrong on the database. This causes:

• Database → auth failure (FATAL: password authentication failed)
• API → misleading connection timeout error
• Worker and Proxy → downstream failures

Agent must: trace root cause (database credentials) rather than patching symptoms. Misleading logs make this hard.

Expected score: ~0.35 (GPT-4o-mini)

---

API Endpoints

Method	Endpoint	Description
POST	/reset?task_id=task1	Start new episode
POST	/step	Submit action, get observation + reward
GET	/state	Current environment state
GET	/tasks	Task list and action schema
GET	/grader?task_id=task1	Score current episode
POST	/baseline	Run baseline agent on all tasks
GET	/health	Health check

---

Setup & Running

Local (Python)

git clone https://github.com/your-username/dev-env-debugger
cd dev-env-debugger
pip install -r requirements.txt
python api.py
# Server starts at http://localhost:7860

Docker

docker build -t dev-env-debugger .
docker run -p 7860:7860 dev-env-debugger

Validate OpenEnv spec

openenv validate

---

Running the Baseline

export OPENAI_API_KEY=your_key_here

# Run all tasks
python baseline.py

# Run single task
python baseline.py --task task1

# JSON output (used by /baseline endpoint)
python baseline.py --output-json

Baseline Scores (GPT-4o-mini)

Task	Difficulty	Expected Score
task1	Easy	~0.85
task2	Medium	~0.60
task3	Hard	~0.35
Average		~0.60

---

Quick API Example

# Start episode
curl -X POST "http://localhost:7860/reset?task_id=task1"

# Read logs
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"type": "read_logs", "service": "api"}'

# Fix env var
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"type": "edit_env", "service": "api", "key": "DB_PORT", "value": "5432"}'

# Restart service
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"type": "restart_service", "service": "api"}'

# Submit
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"type": "submit"}'

# Get score
curl "http://localhost:7860/grader?task_id=task1"

---

Project Structure

dev-env-debugger/
├── environment.py   # Core simulation — state machine, fault injection, action handlers
├── tasks.py         # Task definitions and programmatic graders
├── models.py        # Pydantic models (Observation, Action, Reward, ServiceStatus)
├── api.py           # FastAPI server exposing all endpoints
├── baseline.py      # LLM baseline inference script
├── openenv.yaml     # OpenEnv metadata
├── requirements.txt
├── Dockerfile
└── README.md

---

License

MIT