Spaces:
Sleeping
Sleeping
Commit ·
5b64237
1
Parent(s): 2930dae
Tests and Openenv
Browse files- Dockerfile +22 -5
- README.md +103 -224
- app.py +52 -203
- data.py +82 -8
- environment.py +357 -185
- graders.py +165 -371
- inference.py +352 -0
- models.py +43 -123
- requirements.txt +3 -1
- test_integration.py +116 -0
- tests_new.py +248 -0
Dockerfile
CHANGED
|
@@ -1,25 +1,42 @@
|
|
| 1 |
# ---------------------------------------------------------------
|
| 2 |
-
#
|
| 3 |
# Space SDK: Docker | Port: 7860
|
| 4 |
# ---------------------------------------------------------------
|
| 5 |
FROM python:3.11-slim
|
| 6 |
|
| 7 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
RUN useradd -m -u 1000 appuser
|
| 9 |
|
| 10 |
WORKDIR /app
|
| 11 |
|
| 12 |
-
# Install dependencies first (layer caching)
|
| 13 |
COPY requirements.txt .
|
| 14 |
RUN pip install --no-cache-dir -r requirements.txt
|
| 15 |
|
| 16 |
-
# Copy
|
| 17 |
COPY --chown=appuser:appuser . .
|
| 18 |
|
|
|
|
|
|
|
|
|
|
| 19 |
USER appuser
|
| 20 |
|
| 21 |
# Expose the port HF Spaces expects
|
| 22 |
EXPOSE 7860
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
# Start the FastAPI server
|
| 25 |
-
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1"]
|
|
|
|
| 1 |
# ---------------------------------------------------------------
|
| 2 |
+
# DevOpsEnv — Hugging Face Spaces Docker container
|
| 3 |
# Space SDK: Docker | Port: 7860
|
| 4 |
# ---------------------------------------------------------------
|
| 5 |
FROM python:3.11-slim
|
| 6 |
|
| 7 |
+
# Install system utilities for DevOps tasks
|
| 8 |
+
RUN apt-get update && apt-get install -y --no-install-recommends \
|
| 9 |
+
nginx \
|
| 10 |
+
docker.io \
|
| 11 |
+
systemctl \
|
| 12 |
+
curl \
|
| 13 |
+
git \
|
| 14 |
+
vim \
|
| 15 |
+
&& rm -rf /var/lib/apt/lists/*
|
| 16 |
+
|
| 17 |
+
# Create non-root user for Hugging Face Spaces
|
| 18 |
RUN useradd -m -u 1000 appuser
|
| 19 |
|
| 20 |
WORKDIR /app
|
| 21 |
|
| 22 |
+
# Install Python dependencies first (layer caching)
|
| 23 |
COPY requirements.txt .
|
| 24 |
RUN pip install --no-cache-dir -r requirements.txt
|
| 25 |
|
| 26 |
+
# Copy application code
|
| 27 |
COPY --chown=appuser:appuser . .
|
| 28 |
|
| 29 |
+
# HF Spaces compatibility
|
| 30 |
+
RUN chmod +x /app/app.py 2>/dev/null || true
|
| 31 |
+
|
| 32 |
USER appuser
|
| 33 |
|
| 34 |
# Expose the port HF Spaces expects
|
| 35 |
EXPOSE 7860
|
| 36 |
|
| 37 |
+
# Health check
|
| 38 |
+
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
|
| 39 |
+
CMD python -c "import requests; requests.get('http://localhost:7860/health')" || exit 1
|
| 40 |
+
|
| 41 |
# Start the FastAPI server
|
| 42 |
+
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860", "--workers", "1", "--reload"]
|
README.md
CHANGED
|
@@ -1,282 +1,161 @@
|
|
| 1 |
-
---
|
| 2 |
-
title:
|
| 3 |
-
emoji:
|
| 4 |
colorFrom: blue
|
| 5 |
-
colorTo:
|
| 6 |
sdk: docker
|
| 7 |
app_port: 7860
|
| 8 |
tags:
|
| 9 |
- openenv
|
| 10 |
-
-
|
| 11 |
-
-
|
| 12 |
-
-
|
| 13 |
- agent-evaluation
|
| 14 |
pinned: false
|
| 15 |
---
|
| 16 |
|
| 17 |
-
#
|
| 18 |
-
|
| 19 |
-
An **OpenEnv-compliant** reinforcement-learning environment where agents learn to resolve real-world SaaS customer support tickets.
|
| 20 |
-
|
| 21 |
-
## Why This Domain?
|
| 22 |
-
|
| 23 |
-
Every software company processes thousands of support tickets daily. Getting the right ticket to the right team with the right priority — and generating quality, empathetic responses — directly impacts revenue and customer satisfaction. SupportEnv provides a realistic, grounded benchmark for evaluating and training support-automation agents.
|
| 24 |
-
|
| 25 |
-
---
|
| 26 |
-
|
| 27 |
-
## Environment Overview
|
| 28 |
-
|
| 29 |
-
**Domain**: SaaS customer support triage
|
| 30 |
-
**Ticket types**: billing disputes, technical failures, account changes, feature requests, compliance requests
|
| 31 |
-
**Dataset**: 15 pre-defined tickets (5 per task) with fully specified ground truth — entirely deterministic, no LLM judge needed
|
| 32 |
-
|
| 33 |
-
---
|
| 34 |
-
|
| 35 |
-
## Observation Space
|
| 36 |
-
|
| 37 |
-
```
|
| 38 |
-
Observation
|
| 39 |
-
├── task_id str "task1" | "task2" | "task3"
|
| 40 |
-
├── task_description str Human-readable task description
|
| 41 |
-
├── episode_id str UUID, used in subsequent /step calls
|
| 42 |
-
├── ticket
|
| 43 |
-
│ ├── ticket_id str
|
| 44 |
-
│ ├── subject str
|
| 45 |
-
│ ├── body str
|
| 46 |
-
│ ├── customer_tier str "free" | "pro" | "enterprise"
|
| 47 |
-
│ ├── account_age_days int
|
| 48 |
-
│ ├── previous_tickets int
|
| 49 |
-
│ └── attachments list[str]
|
| 50 |
-
├── thread_history list[dict] Prior agent actions this episode
|
| 51 |
-
├── available_actions list[str] Valid action_type values right now
|
| 52 |
-
├── step_number int
|
| 53 |
-
├── max_steps int
|
| 54 |
-
└── hint str | null Optional step-level guidance
|
| 55 |
-
```
|
| 56 |
-
|
| 57 |
-
---
|
| 58 |
-
|
| 59 |
-
## Action Space
|
| 60 |
|
| 61 |
-
|
| 62 |
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
| `respond` | `response_text`, `resolution_steps` | Task 3 |
|
| 68 |
-
| `resolve` | `response_text`, `resolution_steps` | Task 3 |
|
| 69 |
-
| `escalate` | `escalation_team`, `escalation_reason` | Task 3 |
|
| 70 |
-
| `submit` | — | All tasks |
|
| 71 |
|
| 72 |
-
|
| 73 |
-
**Priority options**: `low` · `medium` · `high` · `critical`
|
| 74 |
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
## Tasks
|
| 78 |
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
|
|
|
| 82 |
|
| 83 |
-
|
| 84 |
-
|-----------|--------|
|
| 85 |
-
| Category correct | 0.50 |
|
| 86 |
-
| Priority correct | 0.40 |
|
| 87 |
-
| Efficiency (≤ 2 classify attempts) | 0.10 |
|
| 88 |
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
-
##
|
| 94 |
-
**max_steps**: 5
|
| 95 |
-
**Objective**: Extract all key entities (IDs, names, amounts, dates, domains) and identify the complete list of required actions.
|
| 96 |
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
| Action coverage | 0.30 |
|
| 101 |
-
| No hallucination (≤ 2 extra keys) | 0.10 |
|
| 102 |
|
| 103 |
-
|
| 104 |
|
| 105 |
-
|
| 106 |
|
| 107 |
-
|
| 108 |
-
**max_steps**: 8
|
| 109 |
-
**Objective**: Write a professional customer-facing response and an ordered list of resolution steps that fully address the ticket.
|
| 110 |
-
|
| 111 |
-
| Criterion | Weight |
|
| 112 |
-
|-----------|--------|
|
| 113 |
-
| Required keyword coverage | 0.30 |
|
| 114 |
-
| Resolution step coverage | 0.30 |
|
| 115 |
-
| Tone compliance (apology / urgency / timeline) | 0.25 |
|
| 116 |
-
| Response length ≥ minimum word count | 0.10 |
|
| 117 |
-
| No empty/trivial steps | 0.05 |
|
| 118 |
-
|
| 119 |
-
---
|
| 120 |
|
| 121 |
-
|
| 122 |
|
| 123 |
-
|
| 124 |
|
| 125 |
-
|
| 126 |
-
step_cost = -0.02 per step (encourages efficiency)
|
| 127 |
-
submit_bonus = +0.05 for explicitly calling submit
|
| 128 |
-
max_step_penalty = -0.10 if episode ends by exhausting max_steps
|
| 129 |
-
grader_bonus = grader_score × 1.0 (0.0–1.0, added at episode end)
|
| 130 |
-
```
|
| 131 |
|
| 132 |
-
|
| 133 |
|
| 134 |
-
|
| 135 |
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
| Method | Endpoint | Description |
|
| 139 |
-
|--------|----------|-------------|
|
| 140 |
-
| `POST` | `/reset` | Start a new episode |
|
| 141 |
-
| `POST` | `/step` | Submit an action |
|
| 142 |
-
| `GET` | `/state` | Get current episode state |
|
| 143 |
-
| `GET` | `/tasks` | List all tasks + action schema |
|
| 144 |
-
| `POST` | `/grader` | Grade a finished episode |
|
| 145 |
-
| `POST` | `/baseline` | Run the heuristic baseline |
|
| 146 |
-
| `GET` | `/health` | Liveness check |
|
| 147 |
-
| `GET` | `/docs` | Interactive Swagger UI |
|
| 148 |
-
|
| 149 |
-
### Quick example
|
| 150 |
-
|
| 151 |
-
```python
|
| 152 |
-
import httpx
|
| 153 |
-
|
| 154 |
-
BASE = "http://localhost:7860"
|
| 155 |
-
|
| 156 |
-
# 1. Start episode
|
| 157 |
-
obs = httpx.post(f"{BASE}/reset", json={"task_id": "task1"}).json()
|
| 158 |
-
episode_id = obs["episode_id"]
|
| 159 |
-
|
| 160 |
-
# 2. Classify the ticket
|
| 161 |
-
result = httpx.post(f"{BASE}/step", json={
|
| 162 |
-
"episode_id": episode_id,
|
| 163 |
-
"action": {
|
| 164 |
-
"action_type": "classify",
|
| 165 |
-
"category": "billing",
|
| 166 |
-
"priority": "high"
|
| 167 |
-
}
|
| 168 |
-
}).json()
|
| 169 |
-
|
| 170 |
-
# 3. Submit
|
| 171 |
-
result = httpx.post(f"{BASE}/step", json={
|
| 172 |
-
"episode_id": episode_id,
|
| 173 |
-
"action": {"action_type": "submit"}
|
| 174 |
-
}).json()
|
| 175 |
-
|
| 176 |
-
# 4. Get grader score
|
| 177 |
-
score = httpx.post(f"{BASE}/grader", json={"episode_id": episode_id}).json()
|
| 178 |
-
print(score["score"]) # e.g. 0.9
|
| 179 |
-
```
|
| 180 |
|
| 181 |
-
|
| 182 |
|
| 183 |
-
##
|
| 184 |
|
| 185 |
-
|
| 186 |
|
| 187 |
-
|
| 188 |
-
git clone <repo-url>
|
| 189 |
-
cd supportenv
|
| 190 |
-
pip install -r requirements.txt
|
| 191 |
-
uvicorn app:app --host 0.0.0.0 --port 7860
|
| 192 |
-
```
|
| 193 |
|
| 194 |
-
##
|
| 195 |
|
| 196 |
-
|
| 197 |
-
docker build -t supportenv .
|
| 198 |
-
docker run -p 7860:7860 supportenv
|
| 199 |
-
```
|
| 200 |
|
| 201 |
-
|
| 202 |
-
|
| 203 |
|
| 204 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 205 |
|
| 206 |
-
|
| 207 |
|
| 208 |
-
##
|
| 209 |
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
python baseline.py --mode heuristic
|
| 213 |
|
| 214 |
-
|
| 215 |
-
python baseline.py --mode heuristic --all-tickets
|
| 216 |
-
```
|
| 217 |
|
| 218 |
-
|
|
|
|
| 219 |
|
| 220 |
-
|
| 221 |
-
export OPENAI_API_KEY="sk-..."
|
| 222 |
-
python baseline.py --mode llm --model gpt-4o-mini
|
| 223 |
-
python baseline.py --mode llm --model gpt-4o-mini --all-tickets
|
| 224 |
-
```
|
| 225 |
|
| 226 |
-
|
| 227 |
|
| 228 |
-
|
| 229 |
|
| 230 |
-
|
| 231 |
|
| 232 |
-
|
| 233 |
-
|------|-------|
|
| 234 |
-
| task1 — Classification (easy) | 1.0000 |
|
| 235 |
-
| task2 — Extraction (medium) | 0.8000 |
|
| 236 |
-
| task3 — Generation (hard) | 1.0000 |
|
| 237 |
-
| **Average** | **0.9333** |
|
| 238 |
|
| 239 |
-
|
|
|
|
|
|
|
| 240 |
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
| task3 — Generation (hard) | 0.9895 | 0.96 – 1.00 |
|
| 246 |
-
| **Overall average** | **0.8036** | |
|
| 247 |
|
| 248 |
-
|
| 249 |
|
| 250 |
-
|
| 251 |
|
| 252 |
-
-
|
| 253 |
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
```
|
| 257 |
-
supportenv/
|
| 258 |
-
├── app.py FastAPI server (all endpoints)
|
| 259 |
-
├── environment.py Episode lifecycle: reset / step / state / grade
|
| 260 |
-
├── graders.py Deterministic graders for all 3 tasks
|
| 261 |
-
├── data.py 15 pre-defined tickets + ground truth
|
| 262 |
-
├── models.py Pydantic typed models
|
| 263 |
-
├── baseline.py Heuristic + LLM baseline inference scripts
|
| 264 |
-
├── openenv.yaml OpenEnv spec metadata
|
| 265 |
-
├── Dockerfile Container (Hugging Face Spaces compatible)
|
| 266 |
-
├── requirements.txt
|
| 267 |
-
└── README.md
|
| 268 |
-
```
|
| 269 |
|
| 270 |
-
-
|
| 271 |
|
| 272 |
-
|
|
|
|
|
|
|
| 273 |
|
| 274 |
-
|
| 275 |
-
- Graders are **purely rule-based** — no LLM judge, identical results every run
|
| 276 |
-
- The heuristic baseline requires **no external API** — scores are fully reproducible
|
| 277 |
|
| 278 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 279 |
|
| 280 |
-
##
|
| 281 |
|
| 282 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: DevOpsEnv
|
| 3 |
+
emoji: 🛠️
|
| 4 |
colorFrom: blue
|
| 5 |
+
colorTo: green
|
| 6 |
sdk: docker
|
| 7 |
app_port: 7860
|
| 8 |
tags:
|
| 9 |
- openenv
|
| 10 |
+
- devops
|
| 11 |
+
- sre
|
| 12 |
+
- troubleshooting
|
| 13 |
- agent-evaluation
|
| 14 |
pinned: false
|
| 15 |
---
|
| 16 |
|
| 17 |
+
# DevOpsEnv
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
|
| 19 |
+
DevOpsEnv is a practice environment where an agent acts like a junior SRE.
|
| 20 |
|
| 21 |
+
In each episode, the agent gets a broken Linux-like system and must fix it by:
|
| 22 |
+
- Running shell commands
|
| 23 |
+
- Editing files
|
| 24 |
+
- Submitting when the fix is done
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
The server gives rewards during the episode and a final score at the end.
|
|
|
|
| 27 |
|
| 28 |
+
## What It Simulates (Simple)
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
There are 3 tasks:
|
| 31 |
+
- Task 1: Nginx is down. Bring service back and verify HTTP is OK.
|
| 32 |
+
- Task 2: Docker compose port mapping is wrong. Fix and redeploy.
|
| 33 |
+
- Task 3: Python API has memory leak behavior. Diagnose and reduce memory usage.
|
| 34 |
|
| 35 |
+
## How It Works
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
+
Step by step:
|
| 38 |
+
1. Call POST /reset with task_id.
|
| 39 |
+
2. You get episode_id plus current system_state.
|
| 40 |
+
3. Call POST /step with an action.
|
| 41 |
+
4. Repeat steps until done, or send action_type submit.
|
| 42 |
+
5. Call POST /grader to get final score and breakdown.
|
| 43 |
|
| 44 |
+
Main endpoints:
|
| 45 |
+
- GET /health
|
| 46 |
+
- GET /tasks
|
| 47 |
+
- POST /reset
|
| 48 |
+
- POST /step
|
| 49 |
+
- GET /state
|
| 50 |
+
- POST /grader
|
| 51 |
|
| 52 |
+
## Action Types
|
|
|
|
|
|
|
| 53 |
|
| 54 |
+
- bash_cmd: Run a command like systemctl status nginx
|
| 55 |
+
- file_edit: Replace content of a file path
|
| 56 |
+
- submit: End episode and grade
|
|
|
|
|
|
|
| 57 |
|
| 58 |
+
## Quick Start (Normal)
|
| 59 |
|
| 60 |
+
### 1) Install
|
| 61 |
|
| 62 |
+
Windows PowerShell:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
+
python -m pip install -r requirements.txt
|
| 65 |
|
| 66 |
+
### 2) Start server
|
| 67 |
|
| 68 |
+
python -m uvicorn app:app --host 0.0.0.0 --port 7860
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
+
### 3) Check health
|
| 71 |
|
| 72 |
+
In another terminal:
|
| 73 |
|
| 74 |
+
Invoke-WebRequest -Uri "http://127.0.0.1:7860/health" -UseBasicParsing
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
+
If working, response includes status: healthy.
|
| 77 |
|
| 78 |
+
### 4) Run built-in integration test
|
| 79 |
|
| 80 |
+
python test_integration.py
|
| 81 |
|
| 82 |
+
If working, you should see all 3 tasks run and a final success message.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
+
## Minimal API Example (Normal)
|
| 85 |
|
| 86 |
+
PowerShell example:
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
+
$reset = Invoke-WebRequest -Uri "http://127.0.0.1:7860/reset" -Method POST -ContentType "application/json" -Body '{"task_id":"task1"}' | Select-Object -ExpandProperty Content | ConvertFrom-Json
|
| 89 |
+
$episodeId = $reset.episode_id
|
| 90 |
|
| 91 |
+
$step = @{
|
| 92 |
+
episode_id = $episodeId
|
| 93 |
+
action = @{
|
| 94 |
+
action_type = "bash_cmd"
|
| 95 |
+
command = "systemctl restart nginx"
|
| 96 |
+
}
|
| 97 |
+
} | ConvertTo-Json -Depth 5
|
| 98 |
|
| 99 |
+
Invoke-WebRequest -Uri "http://127.0.0.1:7860/step" -Method POST -ContentType "application/json" -Body $step
|
| 100 |
|
| 101 |
+
## Test With LLM (OpenAI Key)
|
| 102 |
|
| 103 |
+
1) Keep API server running.
|
| 104 |
+
2) Set key and run inference:
|
|
|
|
| 105 |
|
| 106 |
+
PowerShell:
|
|
|
|
|
|
|
| 107 |
|
| 108 |
+
$env:OPENAI_API_KEY = "your-openai-key"
|
| 109 |
+
python inference.py --task task1 --model gpt-4o-mini
|
| 110 |
|
| 111 |
+
You should see step logs, rewards, and a grader score.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
+
## Test With Gemini API Key
|
| 114 |
|
| 115 |
+
inference.py now supports OpenAI-compatible base URLs.
|
| 116 |
|
| 117 |
+
Use Gemini via OpenAI-compatible endpoint:
|
| 118 |
|
| 119 |
+
PowerShell:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
|
| 121 |
+
$env:GEMINI_API_KEY = "your-gemini-key"
|
| 122 |
+
$env:OPENAI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/openai/"
|
| 123 |
+
python inference.py --task task1 --model gemini-2.5-flash
|
| 124 |
|
| 125 |
+
Notes:
|
| 126 |
+
- You can also use OPENAI_API_KEY instead of GEMINI_API_KEY.
|
| 127 |
+
- If your model name is unavailable, switch to a Gemini model enabled on your key.
|
| 128 |
+
- Keep the environment server running at http://127.0.0.1:7860 (or pass --api-url).
|
|
|
|
|
|
|
| 129 |
|
| 130 |
+
## Docker
|
| 131 |
|
| 132 |
+
Build:
|
| 133 |
|
| 134 |
+
docker build -t devopsenv .
|
| 135 |
|
| 136 |
+
Run:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
+
docker run -p 7860:7860 devopsenv
|
| 139 |
|
| 140 |
+
Then open:
|
| 141 |
+
- http://127.0.0.1:7860/health
|
| 142 |
+
- http://127.0.0.1:7860/docs
|
| 143 |
|
| 144 |
+
## Project Files
|
|
|
|
|
|
|
| 145 |
|
| 146 |
+
- app.py: FastAPI API
|
| 147 |
+
- environment.py: episode logic and simulator
|
| 148 |
+
- graders.py: deterministic scoring
|
| 149 |
+
- data.py: task metadata
|
| 150 |
+
- models.py: Pydantic schemas
|
| 151 |
+
- inference.py: LLM baseline runner
|
| 152 |
+
- test_integration.py: local end-to-end check
|
| 153 |
|
| 154 |
+
## Troubleshooting
|
| 155 |
|
| 156 |
+
- Port already in use:
|
| 157 |
+
- change server port or stop old process.
|
| 158 |
+
- 400/404 from API:
|
| 159 |
+
- check episode_id and task_id values.
|
| 160 |
+
- LLM errors:
|
| 161 |
+
- verify API key, model name, and OPENAI_BASE_URL for Gemini.
|
app.py
CHANGED
|
@@ -1,24 +1,23 @@
|
|
| 1 |
"""
|
| 2 |
-
|
| 3 |
|
| 4 |
-
Endpoints
|
| 5 |
---------
|
| 6 |
POST /reset Create a new episode
|
| 7 |
POST /step Advance the episode
|
| 8 |
GET /state Current episode state
|
| 9 |
GET /tasks List tasks and action schema
|
| 10 |
POST /grader Grade a finished episode
|
| 11 |
-
POST /baseline Run the built-in baseline agent on all tasks
|
| 12 |
GET /health Liveness check
|
| 13 |
GET / Info / spec link
|
| 14 |
"""
|
| 15 |
from __future__ import annotations
|
| 16 |
|
| 17 |
import os
|
| 18 |
-
import
|
| 19 |
-
import
|
| 20 |
-
import tempfile
|
| 21 |
from typing import Any, Dict, List, Optional
|
|
|
|
| 22 |
|
| 23 |
from fastapi import FastAPI, HTTPException, Query
|
| 24 |
from fastapi.middleware.cors import CORSMiddleware
|
|
@@ -28,21 +27,17 @@ import environment as env
|
|
| 28 |
from data import TASK_META
|
| 29 |
from models import (
|
| 30 |
Action,
|
| 31 |
-
BaselineResult,
|
| 32 |
-
GraderResponse,
|
| 33 |
Observation,
|
| 34 |
State,
|
| 35 |
StepResult,
|
| 36 |
TaskInfo,
|
|
|
|
|
|
|
| 37 |
)
|
| 38 |
|
| 39 |
app = FastAPI(
|
| 40 |
-
title="
|
| 41 |
-
description=
|
| 42 |
-
"An OpenEnv-compliant customer-support triage environment. "
|
| 43 |
-
"Agents learn to classify, extract information from, and resolve "
|
| 44 |
-
"real-world SaaS support tickets."
|
| 45 |
-
),
|
| 46 |
version="1.0.0",
|
| 47 |
docs_url="/docs",
|
| 48 |
redoc_url="/redoc",
|
|
@@ -56,13 +51,8 @@ app.add_middleware(
|
|
| 56 |
)
|
| 57 |
|
| 58 |
|
| 59 |
-
# ---------------------------------------------------------------------------
|
| 60 |
-
# Request / response shapes for endpoints not covered by models.py
|
| 61 |
-
# ---------------------------------------------------------------------------
|
| 62 |
-
|
| 63 |
class ResetRequest(BaseModel):
|
| 64 |
task_id: str
|
| 65 |
-
ticket_index: Optional[int] = None
|
| 66 |
|
| 67 |
|
| 68 |
class StepRequest(BaseModel):
|
|
@@ -81,10 +71,10 @@ class GraderRequest(BaseModel):
|
|
| 81 |
@app.get("/", tags=["meta"])
|
| 82 |
def root():
|
| 83 |
return {
|
| 84 |
-
"name": "
|
| 85 |
"version": "1.0.0",
|
| 86 |
-
"description": "OpenEnv
|
| 87 |
-
"openenv_spec": "https://github.com/
|
| 88 |
"tasks": list(TASK_META.keys()),
|
| 89 |
"endpoints": {
|
| 90 |
"reset": "POST /reset",
|
|
@@ -92,7 +82,6 @@ def root():
|
|
| 92 |
"state": "GET /state?episode_id=...",
|
| 93 |
"tasks": "GET /tasks",
|
| 94 |
"grader": "POST /grader",
|
| 95 |
-
"baseline": "POST /baseline",
|
| 96 |
"health": "GET /health",
|
| 97 |
"docs": "GET /docs",
|
| 98 |
},
|
|
@@ -101,140 +90,11 @@ def root():
|
|
| 101 |
|
| 102 |
@app.get("/health", tags=["meta"])
|
| 103 |
def health():
|
| 104 |
-
return {"status": "
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
# ---------------------------------------------------------------------------
|
| 108 |
-
# Core OpenEnv endpoints
|
| 109 |
-
# ---------------------------------------------------------------------------
|
| 110 |
-
|
| 111 |
-
@app.post("/reset", response_model=Observation, tags=["openenv"])
|
| 112 |
-
def reset(request: ResetRequest) -> Observation:
|
| 113 |
-
"""
|
| 114 |
-
Start a new episode.
|
| 115 |
-
|
| 116 |
-
- **task_id**: `task1` | `task2` | `task3`
|
| 117 |
-
- **ticket_index**: 0-indexed ticket to use (optional; default 0)
|
| 118 |
-
"""
|
| 119 |
-
try:
|
| 120 |
-
return env.reset(request.task_id, request.ticket_index)
|
| 121 |
-
except ValueError as e:
|
| 122 |
-
raise HTTPException(status_code=400, detail=str(e))
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
@app.post("/step", response_model=StepResult, tags=["openenv"])
|
| 126 |
-
def step(request: StepRequest) -> StepResult:
|
| 127 |
-
"""
|
| 128 |
-
Submit an action and advance the episode.
|
| 129 |
-
|
| 130 |
-
The `action` object must include `action_type` and the fields relevant
|
| 131 |
-
to that action type (see GET /tasks for the schema).
|
| 132 |
-
"""
|
| 133 |
-
try:
|
| 134 |
-
return env.step(request.episode_id, request.action)
|
| 135 |
-
except KeyError:
|
| 136 |
-
raise HTTPException(
|
| 137 |
-
status_code=404,
|
| 138 |
-
detail=f"Episode '{request.episode_id}' not found. Call POST /reset first.",
|
| 139 |
-
)
|
| 140 |
-
except ValueError as e:
|
| 141 |
-
raise HTTPException(status_code=400, detail=str(e))
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
@app.get("/state", response_model=State, tags=["openenv"])
|
| 145 |
-
def state(episode_id: str = Query(..., description="Episode UUID from POST /reset")) -> State:
|
| 146 |
-
"""Return the current state of an episode."""
|
| 147 |
-
try:
|
| 148 |
-
return env.state(episode_id)
|
| 149 |
-
except KeyError:
|
| 150 |
-
raise HTTPException(
|
| 151 |
-
status_code=404,
|
| 152 |
-
detail=f"Episode '{episode_id}' not found.",
|
| 153 |
-
)
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
# ---------------------------------------------------------------------------
|
| 157 |
-
# /tasks — task listing + action schema
|
| 158 |
-
# ---------------------------------------------------------------------------
|
| 159 |
-
|
| 160 |
-
# JSON Schema for the Action model (subset used in each task)
|
| 161 |
-
_BASE_ACTION_SCHEMA = {
|
| 162 |
-
"type": "object",
|
| 163 |
-
"required": ["action_type"],
|
| 164 |
-
"properties": {
|
| 165 |
-
"action_type": {
|
| 166 |
-
"type": "string",
|
| 167 |
-
"description": "One of the available_actions listed in the Observation",
|
| 168 |
-
},
|
| 169 |
-
},
|
| 170 |
-
}
|
| 171 |
-
|
| 172 |
-
_ACTION_SCHEMAS: Dict[str, Dict[str, Any]] = {
|
| 173 |
-
"task1": {
|
| 174 |
-
**_BASE_ACTION_SCHEMA,
|
| 175 |
-
"description": "classify action: set category + priority; then submit",
|
| 176 |
-
"properties": {
|
| 177 |
-
**_BASE_ACTION_SCHEMA["properties"],
|
| 178 |
-
"category": {
|
| 179 |
-
"type": "string",
|
| 180 |
-
"enum": [
|
| 181 |
-
"billing", "technical", "account",
|
| 182 |
-
"feature_request", "complaint", "general",
|
| 183 |
-
],
|
| 184 |
-
},
|
| 185 |
-
"priority": {
|
| 186 |
-
"type": "string",
|
| 187 |
-
"enum": ["low", "medium", "high", "critical"],
|
| 188 |
-
},
|
| 189 |
-
},
|
| 190 |
-
},
|
| 191 |
-
"task2": {
|
| 192 |
-
**_BASE_ACTION_SCHEMA,
|
| 193 |
-
"description": "extract action: populate extracted_entities + required_actions; then submit",
|
| 194 |
-
"properties": {
|
| 195 |
-
**_BASE_ACTION_SCHEMA["properties"],
|
| 196 |
-
"extracted_entities": {
|
| 197 |
-
"type": "object",
|
| 198 |
-
"additionalProperties": True,
|
| 199 |
-
"description": "Key-value pairs extracted from the ticket text",
|
| 200 |
-
},
|
| 201 |
-
"required_actions": {
|
| 202 |
-
"type": "array",
|
| 203 |
-
"items": {"type": "string"},
|
| 204 |
-
"description": "List of action identifiers (snake_case) needed to close the ticket",
|
| 205 |
-
},
|
| 206 |
-
},
|
| 207 |
-
},
|
| 208 |
-
"task3": {
|
| 209 |
-
**_BASE_ACTION_SCHEMA,
|
| 210 |
-
"description": (
|
| 211 |
-
"respond or resolve action: write response_text + resolution_steps; "
|
| 212 |
-
"optionally escalate; then submit"
|
| 213 |
-
),
|
| 214 |
-
"properties": {
|
| 215 |
-
**_BASE_ACTION_SCHEMA["properties"],
|
| 216 |
-
"response_text": {
|
| 217 |
-
"type": "string",
|
| 218 |
-
"description": "Full professional response to send to the customer",
|
| 219 |
-
},
|
| 220 |
-
"resolution_steps": {
|
| 221 |
-
"type": "array",
|
| 222 |
-
"items": {"type": "string"},
|
| 223 |
-
"description": "Ordered steps for support staff to resolve the ticket",
|
| 224 |
-
},
|
| 225 |
-
"escalation_team": {
|
| 226 |
-
"type": "string",
|
| 227 |
-
"enum": ["billing_team", "engineering", "account_management", "legal"],
|
| 228 |
-
},
|
| 229 |
-
"escalation_reason": {"type": "string"},
|
| 230 |
-
},
|
| 231 |
-
},
|
| 232 |
-
}
|
| 233 |
|
| 234 |
|
| 235 |
-
@app.get("/tasks",
|
| 236 |
-
def
|
| 237 |
-
"""Return metadata and action schema for all tasks."""
|
| 238 |
result = []
|
| 239 |
for task_id, meta in TASK_META.items():
|
| 240 |
result.append(
|
|
@@ -244,66 +104,55 @@ def list_tasks() -> List[TaskInfo]:
|
|
| 244 |
description=meta["description"],
|
| 245 |
difficulty=meta["difficulty"],
|
| 246 |
max_steps=meta["max_steps"],
|
| 247 |
-
action_schema=_ACTION_SCHEMAS[task_id],
|
| 248 |
)
|
| 249 |
)
|
| 250 |
return result
|
| 251 |
|
| 252 |
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
# ---------------------------------------------------------------------------
|
| 256 |
-
|
| 257 |
-
@app.post("/grader", response_model=GraderResponse, tags=["openenv"])
|
| 258 |
-
def grader(request: GraderRequest) -> GraderResponse:
|
| 259 |
-
"""
|
| 260 |
-
Grade a finished episode.
|
| 261 |
-
|
| 262 |
-
The episode must have reached `done=True` (either via a `submit` action
|
| 263 |
-
or by exhausting `max_steps`).
|
| 264 |
-
"""
|
| 265 |
try:
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
raise HTTPException(
|
| 269 |
-
status_code=404,
|
| 270 |
-
detail=f"Episode '{request.episode_id}' not found.",
|
| 271 |
-
)
|
| 272 |
except ValueError as e:
|
| 273 |
raise HTTPException(status_code=400, detail=str(e))
|
| 274 |
|
| 275 |
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 279 |
|
| 280 |
-
class BaselineRequest(BaseModel):
|
| 281 |
-
model: str = "gpt-4o-mini"
|
| 282 |
-
ticket_index: Optional[int] = 0
|
| 283 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
|
| 285 |
-
@app.post("/baseline", response_model=BaselineResult, tags=["openenv"])
|
| 286 |
-
def run_baseline(request: BaselineRequest) -> BaselineResult:
|
| 287 |
-
"""
|
| 288 |
-
Run the heuristic baseline agent against all three tasks.
|
| 289 |
|
| 290 |
-
|
| 291 |
-
|
| 292 |
-
If you want to run the LLM baseline, call `baseline.py` directly.
|
| 293 |
-
"""
|
| 294 |
try:
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
|
| 309 |
-
|
|
|
|
|
|
|
|
|
| 1 |
"""
|
| 2 |
+
FastAPI server for DevOpsEnv - Linux DevOps/SRE troubleshooting environment.
|
| 3 |
|
| 4 |
+
Endpoints:
|
| 5 |
---------
|
| 6 |
POST /reset Create a new episode
|
| 7 |
POST /step Advance the episode
|
| 8 |
GET /state Current episode state
|
| 9 |
GET /tasks List tasks and action schema
|
| 10 |
POST /grader Grade a finished episode
|
|
|
|
| 11 |
GET /health Liveness check
|
| 12 |
GET / Info / spec link
|
| 13 |
"""
|
| 14 |
from __future__ import annotations
|
| 15 |
|
| 16 |
import os
|
| 17 |
+
import uuid
|
| 18 |
+
import json
|
|
|
|
| 19 |
from typing import Any, Dict, List, Optional
|
| 20 |
+
from datetime import datetime
|
| 21 |
|
| 22 |
from fastapi import FastAPI, HTTPException, Query
|
| 23 |
from fastapi.middleware.cors import CORSMiddleware
|
|
|
|
| 27 |
from data import TASK_META
|
| 28 |
from models import (
|
| 29 |
Action,
|
|
|
|
|
|
|
| 30 |
Observation,
|
| 31 |
State,
|
| 32 |
StepResult,
|
| 33 |
TaskInfo,
|
| 34 |
+
Reward,
|
| 35 |
+
GraderResponse,
|
| 36 |
)
|
| 37 |
|
| 38 |
app = FastAPI(
|
| 39 |
+
title="DevOpsEnv",
|
| 40 |
+
description="An OpenEnv-compliant Linux DevOps/SRE troubleshooting environment.",
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
version="1.0.0",
|
| 42 |
docs_url="/docs",
|
| 43 |
redoc_url="/redoc",
|
|
|
|
| 51 |
)
|
| 52 |
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
class ResetRequest(BaseModel):
|
| 55 |
task_id: str
|
|
|
|
| 56 |
|
| 57 |
|
| 58 |
class StepRequest(BaseModel):
|
|
|
|
| 71 |
@app.get("/", tags=["meta"])
|
| 72 |
def root():
|
| 73 |
return {
|
| 74 |
+
"name": "DevOpsEnv",
|
| 75 |
"version": "1.0.0",
|
| 76 |
+
"description": "OpenEnv DevOps/SRE troubleshooting environment",
|
| 77 |
+
"openenv_spec": "https://github.com/meta-pytorch/OpenEnv",
|
| 78 |
"tasks": list(TASK_META.keys()),
|
| 79 |
"endpoints": {
|
| 80 |
"reset": "POST /reset",
|
|
|
|
| 82 |
"state": "GET /state?episode_id=...",
|
| 83 |
"tasks": "GET /tasks",
|
| 84 |
"grader": "POST /grader",
|
|
|
|
| 85 |
"health": "GET /health",
|
| 86 |
"docs": "GET /docs",
|
| 87 |
},
|
|
|
|
| 90 |
|
| 91 |
@app.get("/health", tags=["meta"])
|
| 92 |
def health():
|
| 93 |
+
return {"status": "healthy", "timestamp": datetime.now().isoformat()}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
|
| 96 |
+
@app.get("/tasks", tags=["tasks"])
|
| 97 |
+
def tasks():
|
|
|
|
| 98 |
result = []
|
| 99 |
for task_id, meta in TASK_META.items():
|
| 100 |
result.append(
|
|
|
|
| 104 |
description=meta["description"],
|
| 105 |
difficulty=meta["difficulty"],
|
| 106 |
max_steps=meta["max_steps"],
|
|
|
|
| 107 |
)
|
| 108 |
)
|
| 109 |
return result
|
| 110 |
|
| 111 |
|
| 112 |
+
@app.post("/reset", tags=["control"])
|
| 113 |
+
def reset(req: ResetRequest):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
try:
|
| 115 |
+
obs = env.reset(req.task_id)
|
| 116 |
+
return obs.model_dump()
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
except ValueError as e:
|
| 118 |
raise HTTPException(status_code=400, detail=str(e))
|
| 119 |
|
| 120 |
|
| 121 |
+
@app.post("/step", tags=["control"])
|
| 122 |
+
def step(req: StepRequest):
|
| 123 |
+
try:
|
| 124 |
+
result = env.step(req.episode_id, req.action)
|
| 125 |
+
return result.model_dump()
|
| 126 |
+
except (KeyError, ValueError) as e:
|
| 127 |
+
raise HTTPException(status_code=400, detail=str(e))
|
| 128 |
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
+
@app.get("/state", tags=["observation"])
|
| 131 |
+
def state(episode_id: str = Query(...)):
|
| 132 |
+
try:
|
| 133 |
+
st = env.get_state(episode_id)
|
| 134 |
+
return st.model_dump()
|
| 135 |
+
except KeyError as e:
|
| 136 |
+
raise HTTPException(status_code=404, detail=str(e))
|
| 137 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 138 |
|
| 139 |
+
@app.post("/grader", tags=["evaluation"])
|
| 140 |
+
def grader(req: GraderRequest):
|
|
|
|
|
|
|
| 141 |
try:
|
| 142 |
+
score, breakdown, feedback = env.grade(req.episode_id)
|
| 143 |
+
return GraderResponse(
|
| 144 |
+
episode_id=req.episode_id,
|
| 145 |
+
task_id=env._EPISODES[req.episode_id]["task_id"],
|
| 146 |
+
score=score,
|
| 147 |
+
breakdown=breakdown,
|
| 148 |
+
feedback=feedback,
|
| 149 |
+
).model_dump()
|
| 150 |
+
except KeyError as e:
|
| 151 |
+
raise HTTPException(status_code=404, detail=str(e))
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
if __name__ == "__main__":
|
| 155 |
+
import uvicorn
|
| 156 |
+
port = int(os.environ.get("PORT", 7860))
|
| 157 |
+
uvicorn.run(app, host="0.0.0.0", port=port, workers=1)
|
| 158 |
+
|
data.py
CHANGED
|
@@ -1,18 +1,92 @@
|
|
| 1 |
"""
|
| 2 |
-
|
| 3 |
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
Task 1 (easy) — Ticket Classification
|
| 8 |
-
Task 2 (medium) — Information Extraction
|
| 9 |
-
Task 3 (hard) — Resolution Generation
|
| 10 |
"""
|
| 11 |
from __future__ import annotations
|
| 12 |
from typing import Any, Dict, List
|
| 13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
# ---------------------------------------------------------------------------
|
| 15 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
# Agent must choose: category + priority
|
| 17 |
# Categories: billing | technical | account | feature_request | complaint | general
|
| 18 |
# Priorities: low | medium | high | critical
|
|
|
|
| 1 |
"""
|
| 2 |
+
Linux DevOps tasks for SRE troubleshooting environment.
|
| 3 |
|
| 4 |
+
Task 1 (easy) — Restart crashed Nginx service
|
| 5 |
+
Task 2 (medium) — Fix Docker container misconfiguration
|
| 6 |
+
Task 3 (hard) — Debug and fix memory leak in mock API
|
|
|
|
|
|
|
|
|
|
| 7 |
"""
|
| 8 |
from __future__ import annotations
|
| 9 |
from typing import Any, Dict, List
|
| 10 |
|
| 11 |
+
# Nginx service configuration
|
| 12 |
+
NGINX_CONFIG_PATH = "/etc/nginx/nginx.conf"
|
| 13 |
+
NGINX_SYSTEMD_PATH = "/etc/systemd/system/nginx.service"
|
| 14 |
+
|
| 15 |
+
# Docker configuration
|
| 16 |
+
DOCKER_COMPOSE_PATH = "/srv/docker-compose.yml"
|
| 17 |
+
|
| 18 |
+
# Mock API code path
|
| 19 |
+
MOCK_API_PATH = "/opt/mockapi/app.py"
|
| 20 |
+
|
| 21 |
+
# ---------------------------------------------------------------------------
|
| 22 |
+
# TASK DEFINITIONS
|
| 23 |
# ---------------------------------------------------------------------------
|
| 24 |
+
|
| 25 |
+
TASK_META: Dict[str, Dict[str, Any]] = {
|
| 26 |
+
"task1": {
|
| 27 |
+
"name": "Restart Nginx Service",
|
| 28 |
+
"description": (
|
| 29 |
+
"Production Nginx service has crashed. Restart the service, "
|
| 30 |
+
"verify the configuration syntax, and ensure the server "
|
| 31 |
+
"returns HTTP 200 on port 80. Failing checklist:\n"
|
| 32 |
+
"1. Restart nginx (systemctl restart nginx)\n"
|
| 33 |
+
"2. Verify config syntax (nginx -t)\n"
|
| 34 |
+
"3. Confirm service is running (systemctl status nginx)\n"
|
| 35 |
+
"4. Check HTTP 200 response (curl http://localhost:80)"
|
| 36 |
+
),
|
| 37 |
+
"difficulty": "easy",
|
| 38 |
+
"max_steps": 10,
|
| 39 |
+
"available_actions": ["bash_cmd", "submit"],
|
| 40 |
+
"passing_conditions": [
|
| 41 |
+
"nginx_running",
|
| 42 |
+
"config_valid",
|
| 43 |
+
"http_200_response",
|
| 44 |
+
],
|
| 45 |
+
},
|
| 46 |
+
"task2": {
|
| 47 |
+
"name": "Fix Docker Container Configuration",
|
| 48 |
+
"description": (
|
| 49 |
+
"A critical microservice container is misconfigured. The port "
|
| 50 |
+
"mapping in docker-compose.yml is broken. Fix the configuration, "
|
| 51 |
+
"redeploy the container, and verify it's accessible on the "
|
| 52 |
+
"correct port.\n"
|
| 53 |
+
"1. Edit docker-compose.yml (fix port mapping)\n"
|
| 54 |
+
"2. Restart containers (docker-compose up -d)\n"
|
| 55 |
+
"3. Verify container is running\n"
|
| 56 |
+
"4. Check service responds on mapped port"
|
| 57 |
+
),
|
| 58 |
+
"difficulty": "medium",
|
| 59 |
+
"max_steps": 15,
|
| 60 |
+
"available_actions": ["bash_cmd", "file_edit", "submit"],
|
| 61 |
+
"passing_conditions": [
|
| 62 |
+
"docker_compose_valid",
|
| 63 |
+
"container_running",
|
| 64 |
+
"port_accessible",
|
| 65 |
+
],
|
| 66 |
+
},
|
| 67 |
+
"task3": {
|
| 68 |
+
"name": "Find and Fix Memory Leak in Mock API",
|
| 69 |
+
"description": (
|
| 70 |
+
"The Python API service is leaking memory and consuming excessive "
|
| 71 |
+
"resources. Diagnose the memory leak in /opt/mockapi/app.py, fix "
|
| 72 |
+
"the offending code, and restart the service without root access.\n"
|
| 73 |
+
"1. Identify the memory leak (check processes, logs)\n"
|
| 74 |
+
"2. Kill the runaway process\n"
|
| 75 |
+
"3. Fix the code in app.py (patch the leak)\n"
|
| 76 |
+
"4. Restart the service as appuser\n"
|
| 77 |
+
"5. Verify memory usage is normal"
|
| 78 |
+
),
|
| 79 |
+
"difficulty": "hard",
|
| 80 |
+
"max_steps": 20,
|
| 81 |
+
"available_actions": ["bash_cmd", "file_edit", "submit"],
|
| 82 |
+
"passing_conditions": [
|
| 83 |
+
"process_killed",
|
| 84 |
+
"code_fixed",
|
| 85 |
+
"service_restarted",
|
| 86 |
+
"memory_normal",
|
| 87 |
+
],
|
| 88 |
+
},
|
| 89 |
+
}
|
| 90 |
# Agent must choose: category + priority
|
| 91 |
# Categories: billing | technical | account | feature_request | complaint | general
|
| 92 |
# Priorities: low | medium | high | critical
|
environment.py
CHANGED
|
@@ -1,166 +1,402 @@
|
|
| 1 |
"""
|
| 2 |
-
Core
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
|
| 4 |
Manages episode lifecycle:
|
| 5 |
reset() → Observation
|
| 6 |
step(action) → StepResult
|
| 7 |
-
|
| 8 |
-
grade() →
|
| 9 |
-
|
| 10 |
-
Sessions are stored in-memory (dict keyed by episode_id).
|
| 11 |
-
For multi-replica deployments replace with Redis or a database.
|
| 12 |
"""
|
| 13 |
from __future__ import annotations
|
| 14 |
|
| 15 |
import uuid
|
| 16 |
-
|
|
|
|
|
|
|
| 17 |
|
| 18 |
-
from data import TASK_META
|
| 19 |
-
from graders import
|
| 20 |
from models import (
|
| 21 |
Action,
|
| 22 |
-
GraderResponse,
|
| 23 |
Observation,
|
| 24 |
Reward,
|
| 25 |
State,
|
| 26 |
StepResult,
|
| 27 |
-
|
| 28 |
)
|
| 29 |
|
| 30 |
-
# In-memory store:
|
| 31 |
_EPISODES: Dict[str, Dict[str, Any]] = {}
|
| 32 |
|
| 33 |
|
| 34 |
# ---------------------------------------------------------------------------
|
| 35 |
-
#
|
| 36 |
# ---------------------------------------------------------------------------
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
-
def _step_reward_task1(action: Action, done: bool) -> tuple[float, str]:
|
| 44 |
-
"""Immediate step reward for task1 episodes."""
|
| 45 |
-
if action.action_type == "classify":
|
| 46 |
-
if action.category and action.priority:
|
| 47 |
-
return 0.05, "Classify action with both fields set."
|
| 48 |
-
return 0.02, "Classify action (partial — missing category or priority)."
|
| 49 |
-
if action.action_type == "submit":
|
| 50 |
-
return _SUBMIT_REWARD, "Episode submitted."
|
| 51 |
-
return _STEP_COST, f"Action '{action.action_type}' not primary for this task."
|
| 52 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
-
def _step_reward_task2(action: Action, done: bool) -> tuple[float, str]:
|
| 55 |
-
"""Immediate step reward for task2 episodes."""
|
| 56 |
-
if action.action_type == "extract":
|
| 57 |
-
n_entities = len(action.extracted_entities or {})
|
| 58 |
-
n_actions = len(action.required_actions or [])
|
| 59 |
-
reward = min(n_entities * 0.01 + n_actions * 0.02, 0.15)
|
| 60 |
-
return reward, f"Extracted {n_entities} entities, {n_actions} actions."
|
| 61 |
-
if action.action_type == "submit":
|
| 62 |
-
return _SUBMIT_REWARD, "Episode submitted."
|
| 63 |
-
return _STEP_COST, f"Action '{action.action_type}' not primary for this task."
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
def _step_reward_task3(action: Action, done: bool) -> tuple[float, str]:
|
| 67 |
-
"""Immediate step reward for task3 episodes."""
|
| 68 |
-
if action.action_type in ("respond", "resolve"):
|
| 69 |
-
text_len = len((action.response_text or "").split())
|
| 70 |
-
steps_n = len(action.resolution_steps or [])
|
| 71 |
-
reward = min(text_len * 0.001 + steps_n * 0.02, 0.20)
|
| 72 |
-
return reward, f"Response: {text_len} words, {steps_n} steps."
|
| 73 |
-
if action.action_type == "escalate":
|
| 74 |
-
return 0.03, "Escalation action recorded."
|
| 75 |
-
if action.action_type == "submit":
|
| 76 |
-
return _SUBMIT_REWARD, "Episode submitted."
|
| 77 |
-
return _STEP_COST, f"Action '{action.action_type}' not primary for this task."
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
|
| 87 |
# ---------------------------------------------------------------------------
|
| 88 |
-
#
|
| 89 |
# ---------------------------------------------------------------------------
|
| 90 |
|
| 91 |
-
def
|
| 92 |
-
"""
|
| 93 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
-
|
| 96 |
-
task_id: "task1" | "task2" | "task3"
|
| 97 |
-
ticket_index: which ticket to use (0-based); if None, defaults to 0.
|
| 98 |
|
| 99 |
-
|
| 100 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
if task_id not in TASK_META:
|
| 102 |
raise ValueError(f"Unknown task_id {task_id!r}. Valid: {list(TASK_META)}")
|
| 103 |
|
| 104 |
meta = TASK_META[task_id]
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
previous_tickets=raw_ticket["previous_tickets"],
|
| 116 |
-
attachments=raw_ticket.get("attachments", []),
|
| 117 |
-
)
|
| 118 |
|
| 119 |
episode_id = str(uuid.uuid4())
|
| 120 |
_EPISODES[episode_id] = {
|
| 121 |
"task_id": task_id,
|
| 122 |
-
"ticket_index": idx,
|
| 123 |
"step_number": 0,
|
| 124 |
"max_steps": meta["max_steps"],
|
| 125 |
"done": False,
|
| 126 |
"total_reward": 0.0,
|
| 127 |
"action_history": [],
|
| 128 |
"final_score": None,
|
|
|
|
| 129 |
}
|
| 130 |
|
|
|
|
|
|
|
| 131 |
return Observation(
|
| 132 |
task_id=task_id,
|
| 133 |
task_description=meta["description"],
|
| 134 |
episode_id=episode_id,
|
| 135 |
-
|
| 136 |
thread_history=[],
|
| 137 |
available_actions=meta["available_actions"],
|
| 138 |
step_number=0,
|
| 139 |
max_steps=meta["max_steps"],
|
| 140 |
-
hint=
|
| 141 |
)
|
| 142 |
|
| 143 |
|
| 144 |
def step(episode_id: str, action: Action) -> StepResult:
|
| 145 |
-
"""
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
KeyError: episode_id not found
|
| 150 |
-
ValueError: episode is already done
|
| 151 |
-
"""
|
| 152 |
-
ep = _EPISODES[episode_id] # raises KeyError if not found
|
| 153 |
|
| 154 |
if ep["done"]:
|
| 155 |
raise ValueError(f"Episode {episode_id} is already done.")
|
| 156 |
|
| 157 |
-
task_id
|
| 158 |
meta = TASK_META[task_id]
|
| 159 |
-
tickets = get_tickets(task_id)
|
| 160 |
|
| 161 |
ep["step_number"] += 1
|
| 162 |
ep["action_history"].append(action.model_dump())
|
| 163 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
# Determine if done
|
| 165 |
done = False
|
| 166 |
if action.action_type == "submit":
|
|
@@ -168,76 +404,59 @@ def step(episode_id: str, action: Action) -> StepResult:
|
|
| 168 |
elif ep["step_number"] >= ep["max_steps"]:
|
| 169 |
done = True
|
| 170 |
|
| 171 |
-
#
|
| 172 |
-
|
| 173 |
-
step_r, explanation = reward_fn(action, done)
|
| 174 |
-
|
| 175 |
-
if done and action.action_type != "submit":
|
| 176 |
-
step_r += _MAX_STEP_PENALTY
|
| 177 |
-
explanation += f" Max steps reached — {_MAX_STEP_PENALTY} penalty."
|
| 178 |
|
| 179 |
-
# Apply
|
| 180 |
-
final_score = None
|
| 181 |
if done:
|
| 182 |
-
final_score,
|
| 183 |
-
task_id, ep["ticket_index"], ep["action_history"]
|
| 184 |
-
)
|
| 185 |
ep["final_score"] = final_score
|
| 186 |
-
bonus = final_score *
|
| 187 |
-
|
| 188 |
-
explanation += f" Grader
|
|
|
|
|
|
|
| 189 |
|
| 190 |
-
ep["total_reward"] = round(ep["total_reward"] +
|
| 191 |
ep["done"] = done
|
| 192 |
|
| 193 |
-
# Build
|
| 194 |
-
|
| 195 |
-
ticket = TicketInfo(
|
| 196 |
-
ticket_id=raw_ticket["ticket_id"],
|
| 197 |
-
subject=raw_ticket["subject"],
|
| 198 |
-
body=raw_ticket["body"],
|
| 199 |
-
customer_tier=raw_ticket["customer_tier"],
|
| 200 |
-
account_age_days=raw_ticket["account_age_days"],
|
| 201 |
-
previous_tickets=raw_ticket["previous_tickets"],
|
| 202 |
-
attachments=raw_ticket.get("attachments", []),
|
| 203 |
-
)
|
| 204 |
-
|
| 205 |
thread_history = [
|
| 206 |
-
{"role": "agent", "content": str(a)}
|
| 207 |
-
for a in ep["action_history"]
|
| 208 |
]
|
| 209 |
|
| 210 |
obs = Observation(
|
| 211 |
task_id=task_id,
|
| 212 |
task_description=meta["description"],
|
| 213 |
episode_id=episode_id,
|
| 214 |
-
|
| 215 |
thread_history=thread_history,
|
| 216 |
available_actions=meta["available_actions"] if not done else [],
|
| 217 |
step_number=ep["step_number"],
|
| 218 |
max_steps=ep["max_steps"],
|
| 219 |
-
hint=
|
| 220 |
)
|
| 221 |
|
| 222 |
reward = Reward(
|
| 223 |
-
step_reward=round(
|
| 224 |
total_reward=ep["total_reward"],
|
| 225 |
explanation=explanation,
|
| 226 |
)
|
| 227 |
|
| 228 |
-
info
|
| 229 |
-
"step": ep["step_number"],
|
| 230 |
-
"max_steps": ep["max_steps"],
|
| 231 |
-
}
|
| 232 |
if done:
|
| 233 |
info["final_score"] = final_score
|
| 234 |
|
| 235 |
return StepResult(observation=obs, reward=reward, done=done, info=info)
|
| 236 |
|
| 237 |
|
| 238 |
-
def
|
| 239 |
"""Return the current state of an episode."""
|
| 240 |
-
ep = _EPISODES
|
|
|
|
|
|
|
|
|
|
| 241 |
return State(
|
| 242 |
task_id=ep["task_id"],
|
| 243 |
episode_id=episode_id,
|
|
@@ -250,65 +469,18 @@ def state(episode_id: str) -> State:
|
|
| 250 |
)
|
| 251 |
|
| 252 |
|
| 253 |
-
def grade(episode_id: str) ->
|
| 254 |
-
"""
|
| 255 |
-
|
|
|
|
|
|
|
| 256 |
|
| 257 |
-
|
| 258 |
-
ValueError
|
| 259 |
-
"""
|
| 260 |
-
ep = _EPISODES[episode_id]
|
| 261 |
-
if not ep["done"]:
|
| 262 |
-
raise ValueError(
|
| 263 |
-
f"Episode {episode_id} is not done yet (step {ep['step_number']}/{ep['max_steps']})."
|
| 264 |
-
)
|
| 265 |
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
)
|
| 269 |
ep["final_score"] = score
|
| 270 |
|
| 271 |
-
return
|
| 272 |
-
episode_id=episode_id,
|
| 273 |
-
task_id=ep["task_id"],
|
| 274 |
-
score=score,
|
| 275 |
-
breakdown=breakdown,
|
| 276 |
-
feedback=feedback,
|
| 277 |
-
)
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
# ---------------------------------------------------------------------------
|
| 281 |
-
# Hints
|
| 282 |
-
# ---------------------------------------------------------------------------
|
| 283 |
-
|
| 284 |
-
_HINTS = {
|
| 285 |
-
"task1": [
|
| 286 |
-
"Read the ticket carefully. What is the customer's main problem?",
|
| 287 |
-
"Review your classification — does the priority match the urgency in the message?",
|
| 288 |
-
"Submit your final answer.",
|
| 289 |
-
],
|
| 290 |
-
"task2": [
|
| 291 |
-
"Scan the ticket for IDs, names, dates, and amounts.",
|
| 292 |
-
"List every action that support staff must take to close this ticket.",
|
| 293 |
-
"Verify you haven't missed any entity.",
|
| 294 |
-
"Double-check required_actions against the ticket body.",
|
| 295 |
-
"Submit your final answer.",
|
| 296 |
-
],
|
| 297 |
-
"task3": [
|
| 298 |
-
"Read the ticket carefully — what is the customer asking for?",
|
| 299 |
-
"Draft a professional response that acknowledges the issue.",
|
| 300 |
-
"Ensure your resolution_steps are actionable and ordered.",
|
| 301 |
-
"Check that your response covers all required information.",
|
| 302 |
-
"Add any missing keywords (e.g. apology, timeline).",
|
| 303 |
-
"Review tone — is it empathetic and professional?",
|
| 304 |
-
"Verify step completeness before submitting.",
|
| 305 |
-
"Submit your final answer.",
|
| 306 |
-
],
|
| 307 |
-
}
|
| 308 |
-
|
| 309 |
|
| 310 |
-
def _get_hint(task_id: str, step_number: int) -> Optional[str]:
|
| 311 |
-
hints = _HINTS.get(task_id, [])
|
| 312 |
-
if step_number < len(hints):
|
| 313 |
-
return hints[step_number]
|
| 314 |
-
return None
|
|
|
|
| 1 |
"""
|
| 2 |
+
Core DevOpsEnv environment logic.
|
| 3 |
+
|
| 4 |
+
Simulates a broken Linux server with:
|
| 5 |
+
- Task 1: Crashed Nginx service needing restart
|
| 6 |
+
- Task 2: Misconfigured Docker container
|
| 7 |
+
- Task 3: Memory leak in Python mock API
|
| 8 |
|
| 9 |
Manages episode lifecycle:
|
| 10 |
reset() → Observation
|
| 11 |
step(action) → StepResult
|
| 12 |
+
get_state() → State
|
| 13 |
+
grade() → (score, breakdown, feedback)
|
|
|
|
|
|
|
|
|
|
| 14 |
"""
|
| 15 |
from __future__ import annotations
|
| 16 |
|
| 17 |
import uuid
|
| 18 |
+
import json
|
| 19 |
+
import re
|
| 20 |
+
from typing import Any, Dict, Optional, Tuple, List
|
| 21 |
|
| 22 |
+
from data import TASK_META
|
| 23 |
+
from graders import grade_task
|
| 24 |
from models import (
|
| 25 |
Action,
|
|
|
|
| 26 |
Observation,
|
| 27 |
Reward,
|
| 28 |
State,
|
| 29 |
StepResult,
|
| 30 |
+
SystemState,
|
| 31 |
)
|
| 32 |
|
| 33 |
+
# In-memory store: episode_id → EpisodeState dict
|
| 34 |
_EPISODES: Dict[str, Dict[str, Any]] = {}
|
| 35 |
|
| 36 |
|
| 37 |
# ---------------------------------------------------------------------------
|
| 38 |
+
# Mock filesystem and system state
|
| 39 |
# ---------------------------------------------------------------------------
|
| 40 |
|
| 41 |
+
def _create_initial_state_task1() -> Dict[str, Any]:
|
| 42 |
+
"""Task 1: Nginx is crashed."""
|
| 43 |
+
return {
|
| 44 |
+
"running_processes": [
|
| 45 |
+
{"pid": 100, "name": "systemd"},
|
| 46 |
+
{"pid": 105, "name": "sshd"},
|
| 47 |
+
# nginx NOT running
|
| 48 |
+
],
|
| 49 |
+
"service_status": {
|
| 50 |
+
"nginx": "inactive",
|
| 51 |
+
"docker": "active",
|
| 52 |
+
"mockapi": "active",
|
| 53 |
+
},
|
| 54 |
+
"http_ports_open": [8080], # 80 is down
|
| 55 |
+
"docker_containers": [],
|
| 56 |
+
"logs": "2026-03-29 01:30:00 nginx crashed\nCore dump detected.\n",
|
| 57 |
+
"files": {
|
| 58 |
+
NGINX_CONFIG_PATH: """
|
| 59 |
+
user nginx;
|
| 60 |
+
worker_processes auto;
|
| 61 |
+
error_log /var/log/nginx/error.log warn;
|
| 62 |
+
pid /var/run/nginx.pid;
|
| 63 |
+
|
| 64 |
+
events {
|
| 65 |
+
worker_connections 1024;
|
| 66 |
+
}
|
| 67 |
|
| 68 |
+
http {
|
| 69 |
+
include /etc/nginx/mime.types;
|
| 70 |
+
default_type application/octet-stream;
|
| 71 |
+
sendfile on;
|
| 72 |
+
keepalive_timeout 65;
|
| 73 |
+
|
| 74 |
+
server {
|
| 75 |
+
listen 80 default_server;
|
| 76 |
+
server_name _;
|
| 77 |
+
location / {
|
| 78 |
+
return 200 "OK\\n";
|
| 79 |
+
}
|
| 80 |
+
}
|
| 81 |
+
}""",
|
| 82 |
+
"/etc/systemd/system/nginx.service": """
|
| 83 |
+
[Unit]
|
| 84 |
+
Description=The NGINX HTTP and reverse proxy server
|
| 85 |
+
After=network.target
|
| 86 |
+
|
| 87 |
+
[Service]
|
| 88 |
+
Type=forking
|
| 89 |
+
PIDFile=/var/run/nginx.pid
|
| 90 |
+
ExecStartPre=/usr/sbin/nginx -t
|
| 91 |
+
ExecStart=/usr/sbin/nginx
|
| 92 |
+
ExecReload=/bin/kill -s HUP $MAINPID
|
| 93 |
+
ExecStop=/bin/kill -s QUIT $MAINPID
|
| 94 |
+
PrivateTmp=true
|
| 95 |
+
|
| 96 |
+
[Install]
|
| 97 |
+
WantedBy=multi-user.target""",
|
| 98 |
+
},
|
| 99 |
+
"cpu_usage": 45.2,
|
| 100 |
+
"memory_usage_mb": 256,
|
| 101 |
+
}
|
| 102 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
+
def _create_initial_state_task2() -> Dict[str, Any]:
|
| 105 |
+
"""Task 2: Docker misconfigured."""
|
| 106 |
+
return {
|
| 107 |
+
"running_processes": [
|
| 108 |
+
{"pid": 100, "name": "systemd"},
|
| 109 |
+
{"pid": 105, "name": "sshd"},
|
| 110 |
+
{"pid": 200, "name": "dockerd"},
|
| 111 |
+
],
|
| 112 |
+
"service_status": {
|
| 113 |
+
"nginx": "active",
|
| 114 |
+
"docker": "active",
|
| 115 |
+
"mockapi": "inactive",
|
| 116 |
+
},
|
| 117 |
+
"http_ports_open": [80],
|
| 118 |
+
"docker_containers": [
|
| 119 |
+
{"id": "abc123", "name": "mockapi-svc", "status": "running", "ports": "8000->3000/tcp"}
|
| 120 |
+
],
|
| 121 |
+
"logs": "docker: port 3000 already in use\n",
|
| 122 |
+
"files": {
|
| 123 |
+
"/srv/docker-compose.yml": """
|
| 124 |
+
version: '3.8'
|
| 125 |
+
services:
|
| 126 |
+
mockapi:
|
| 127 |
+
image: mockapi:latest
|
| 128 |
+
ports:
|
| 129 |
+
- "3000:3000"
|
| 130 |
+
environment:
|
| 131 |
+
- PORT=3000
|
| 132 |
+
volumes:
|
| 133 |
+
- ./app.py:/app/app.py""",
|
| 134 |
+
},
|
| 135 |
+
"cpu_usage": 62.0,
|
| 136 |
+
"memory_usage_mb": 1024,
|
| 137 |
+
}
|
| 138 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
|
| 140 |
+
def _create_initial_state_task3() -> Dict[str, Any]:
|
| 141 |
+
"""Task 3: Memory leak in mock API."""
|
| 142 |
+
return {
|
| 143 |
+
"running_processes": [
|
| 144 |
+
{"pid": 100, "name": "systemd"},
|
| 145 |
+
{"pid": 105, "name": "sshd"},
|
| 146 |
+
{"pid": 300, "name": "python3", "rss_mb": 2048, "user": "appuser"}, # MEMORY LEAK
|
| 147 |
+
],
|
| 148 |
+
"service_status": {
|
| 149 |
+
"nginx": "active",
|
| 150 |
+
"docker": "active",
|
| 151 |
+
"mockapi": "active",
|
| 152 |
+
},
|
| 153 |
+
"http_ports_open": [80, 5000],
|
| 154 |
+
"docker_containers": [],
|
| 155 |
+
"logs": (
|
| 156 |
+
"2026-03-29 01:45:00 mockapi started\n"
|
| 157 |
+
"2026-03-29 01:46:00 memory usage: 512 MB\n"
|
| 158 |
+
"2026-03-29 01:47:00 memory usage: 1024 MB\n"
|
| 159 |
+
"2026-03-29 01:48:00 memory usage: 1536 MB (WARNING: HIGH)\n"
|
| 160 |
+
"2026-03-29 01:49:00 memory usage: 2048 MB (CRITICAL)\n"
|
| 161 |
+
),
|
| 162 |
+
"files": {
|
| 163 |
+
"/opt/mockapi/app.py": """
|
| 164 |
+
import json
|
| 165 |
+
from flask import Flask
|
| 166 |
+
|
| 167 |
+
app = Flask(__name__)
|
| 168 |
+
|
| 169 |
+
# BUG: This list grows unbounded
|
| 170 |
+
request_cache = []
|
| 171 |
+
|
| 172 |
+
@app.route('/api/data', methods=['GET'])
|
| 173 |
+
def get_data():
|
| 174 |
+
data = {"timestamp": 123456, "value": 42}
|
| 175 |
+
request_cache.append(data) # MEMORY LEAK!
|
| 176 |
+
return json.dumps(data)
|
| 177 |
+
|
| 178 |
+
if __name__ == '__main__':
|
| 179 |
+
app.run(host='0.0.0.0', port=5000)
|
| 180 |
+
""",
|
| 181 |
+
},
|
| 182 |
+
"cpu_usage": 85.5,
|
| 183 |
+
"memory_usage_mb": 2048,
|
| 184 |
+
}
|
| 185 |
|
| 186 |
+
|
| 187 |
+
NGINX_CONFIG_PATH = "/etc/nginx/nginx.conf"
|
| 188 |
+
DOCKER_COMPOSE_PATH = "/srv/docker-compose.yml"
|
| 189 |
+
MOCK_API_PATH = "/opt/mockapi/app.py"
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def _build_system_state(task_id: str, ep_state: Dict[str, Any]) -> SystemState:
|
| 193 |
+
"""Build a SystemState object from episode state."""
|
| 194 |
+
state_dict = ep_state["system_state"]
|
| 195 |
+
return SystemState(
|
| 196 |
+
task_id=task_id,
|
| 197 |
+
available_commands=["systemctl", "nginx", "docker", "curl", "ps", "cat", "vim"],
|
| 198 |
+
filesystem_snapshot=json.dumps({
|
| 199 |
+
k: v for k, v in state_dict.get("files", {}).items()
|
| 200 |
+
}),
|
| 201 |
+
running_processes=state_dict.get("running_processes", []),
|
| 202 |
+
service_status=state_dict.get("service_status", {}),
|
| 203 |
+
logs=state_dict.get("logs", ""),
|
| 204 |
+
http_ports_open=state_dict.get("http_ports_open", []),
|
| 205 |
+
docker_containers=state_dict.get("docker_containers", []),
|
| 206 |
+
cpu_usage=state_dict.get("cpu_usage", 0.0),
|
| 207 |
+
memory_usage_mb=state_dict.get("memory_usage_mb", 0),
|
| 208 |
+
)
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
# ---------------------------------------------------------------------------
|
| 212 |
+
# Dynamic execution simulation
|
| 213 |
+
# ---------------------------------------------------------------------------
|
| 214 |
+
|
| 215 |
+
def _simulate_bash_cmd(cmd: str, task_id: str, ep_state: Dict[str, Any]) -> str:
|
| 216 |
+
"""Simulate bash command execution."""
|
| 217 |
+
state_dict = ep_state["system_state"]
|
| 218 |
+
lower_cmd = cmd.lower()
|
| 219 |
+
|
| 220 |
+
# Task 1: Nginx commands
|
| 221 |
+
if task_id == "task1":
|
| 222 |
+
if "systemctl restart nginx" in lower_cmd or "systemctl start nginx" in lower_cmd:
|
| 223 |
+
state_dict["service_status"]["nginx"] = "active"
|
| 224 |
+
state_dict["running_processes"].append({"pid": 999, "name": "nginx"})
|
| 225 |
+
state_dict["http_ports_open"] = [80]
|
| 226 |
+
return "Job for nginx.service started successfully."
|
| 227 |
+
elif "systemctl status nginx" in lower_cmd:
|
| 228 |
+
if state_dict["service_status"]["nginx"] == "active":
|
| 229 |
+
return "● nginx.service - NGINX HTTP Server\n Loaded: loaded (/etc/systemd/system/nginx.service)\n Active: active (running)"
|
| 230 |
+
return "● nginx.service - NGINX HTTP Server\n Active: inactive (dead)"
|
| 231 |
+
elif "nginx -t" in lower_cmd:
|
| 232 |
+
return "nginx: the configuration file /etc/nginx/nginx.conf syntax is ok\nnginx: configuration file /etc/nginx/nginx.conf test is successful"
|
| 233 |
+
elif "curl http://localhost:80" in lower_cmd or "curl http://localhost" in lower_cmd:
|
| 234 |
+
if 80 in state_dict["http_ports_open"]:
|
| 235 |
+
return "OK"
|
| 236 |
+
return "curl: (7) Failed to connect to localhost port 80: Connection refused"
|
| 237 |
+
|
| 238 |
+
# Task 2: Docker commands
|
| 239 |
+
elif task_id == "task2":
|
| 240 |
+
if "docker-compose up -d" in lower_cmd:
|
| 241 |
+
if DOCKER_COMPOSE_PATH in state_dict["files"]:
|
| 242 |
+
compose_content = state_dict["files"][DOCKER_COMPOSE_PATH]
|
| 243 |
+
# Check if port is now correct
|
| 244 |
+
if "3000:3000" in compose_content:
|
| 245 |
+
state_dict["docker_containers"] = [
|
| 246 |
+
{"id": "xyz789", "name": "mockapi-svc", "status": "running", "ports": "3000:3000/tcp"}
|
| 247 |
+
]
|
| 248 |
+
state_dict["service_status"]["mockapi"] = "active"
|
| 249 |
+
return "Creating mockapi ... done"
|
| 250 |
+
return "ERROR: docker-compose.yml not found or invalid"
|
| 251 |
+
elif "docker ps" in lower_cmd:
|
| 252 |
+
if state_dict["docker_containers"]:
|
| 253 |
+
return "\n".join([f"{c['id']} {c['name']} {c['status']}" for c in state_dict["docker_containers"]])
|
| 254 |
+
return "No containers running"
|
| 255 |
+
|
| 256 |
+
# Task 3: Process/memory commands
|
| 257 |
+
elif task_id == "task3":
|
| 258 |
+
if "ps aux" in lower_cmd or "ps aux grep python" in lower_cmd:
|
| 259 |
+
output = ""
|
| 260 |
+
for proc in state_dict["running_processes"]:
|
| 261 |
+
if proc.get("name") == "python3":
|
| 262 |
+
output += f"appuser {proc['pid']} 85.5 {proc.get('rss_mb', 512)} python3 /opt/mockapi/app.py\n"
|
| 263 |
+
return output if output else "No python processes found"
|
| 264 |
+
elif "kill" in lower_cmd:
|
| 265 |
+
if "300" in lower_cmd or "python" in lower_cmd:
|
| 266 |
+
state_dict["running_processes"] = [p for p in state_dict["running_processes"] if p.get("name") != "python3"]
|
| 267 |
+
state_dict["service_status"]["mockapi"] = "inactive"
|
| 268 |
+
return "Process killed"
|
| 269 |
+
return "Process not found"
|
| 270 |
+
elif "python3 /opt/mockapi/app.py &" in lower_cmd or "python3 /opt/mockapi/app.py" in lower_cmd:
|
| 271 |
+
state_dict["running_processes"].append({"pid": 301, "name": "python3", "rss_mb": 128, "user": "appuser"})
|
| 272 |
+
state_dict["service_status"]["mockapi"] = "active"
|
| 273 |
+
state_dict["http_ports_open"] = [80, 5000]
|
| 274 |
+
return "Application started"
|
| 275 |
+
|
| 276 |
+
return f"Command '{cmd}' executed (simulated)"
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
def _simulate_file_edit(file_path: str, new_content: str, ep_state: Dict[str, Any]) -> str:
|
| 280 |
+
"""Simulate file editing."""
|
| 281 |
+
state_dict = ep_state["system_state"]
|
| 282 |
+
|
| 283 |
+
if file_path not in state_dict.get("files", {}):
|
| 284 |
+
return f"ERROR: File {file_path} not found"
|
| 285 |
+
|
| 286 |
+
# Detect task 2: Check docker-compose.yml fix
|
| 287 |
+
if file_path == DOCKER_COMPOSE_PATH and "3000:3000" in new_content:
|
| 288 |
+
state_dict["files"][file_path] = new_content
|
| 289 |
+
return f"File {file_path} updated successfully"
|
| 290 |
+
|
| 291 |
+
# Detect task 3: Check mock API fix
|
| 292 |
+
elif file_path == MOCK_API_PATH and "request_cache = []" not in new_content:
|
| 293 |
+
# Verify fix removes the memory leak
|
| 294 |
+
state_dict["files"][file_path] = new_content
|
| 295 |
+
return f"File {file_path} patched successfully"
|
| 296 |
+
|
| 297 |
+
state_dict["files"][file_path] = new_content
|
| 298 |
+
return f"File {file_path} edited"
|
| 299 |
|
| 300 |
|
| 301 |
# ---------------------------------------------------------------------------
|
| 302 |
+
# Reward calculation
|
| 303 |
# ---------------------------------------------------------------------------
|
| 304 |
|
| 305 |
+
def _calculate_step_reward(task_id: str, action: Action, ep_state: Dict[str, Any]) -> Tuple[float, str]:
|
| 306 |
+
"""Calculate reward based on action and task."""
|
| 307 |
+
base_step_cost = -0.01
|
| 308 |
+
reward = base_step_cost
|
| 309 |
+
|
| 310 |
+
if action.action_type == "bash_cmd":
|
| 311 |
+
cmd = action.command or ""
|
| 312 |
+
reward += 0.05
|
| 313 |
+
explanation = f"Executed: {cmd[:50]}"
|
| 314 |
+
return reward, explanation
|
| 315 |
+
|
| 316 |
+
elif action.action_type == "file_edit":
|
| 317 |
+
reward += 0.03
|
| 318 |
+
explanation = f"Edited: {action.file_path}"
|
| 319 |
+
return reward, explanation
|
| 320 |
+
|
| 321 |
+
elif action.action_type == "submit":
|
| 322 |
+
reward += 0.1
|
| 323 |
+
explanation = "Episode submitted for grading"
|
| 324 |
+
return reward, explanation
|
| 325 |
|
| 326 |
+
return reward, "Step taken"
|
|
|
|
|
|
|
| 327 |
|
| 328 |
+
|
| 329 |
+
# ---------------------------------------------------------------------------
|
| 330 |
+
# Core API functions
|
| 331 |
+
# ---------------------------------------------------------------------------
|
| 332 |
+
|
| 333 |
+
def reset(task_id: str) -> Observation:
|
| 334 |
+
"""Create a new episode for the given task."""
|
| 335 |
if task_id not in TASK_META:
|
| 336 |
raise ValueError(f"Unknown task_id {task_id!r}. Valid: {list(TASK_META)}")
|
| 337 |
|
| 338 |
meta = TASK_META[task_id]
|
| 339 |
+
|
| 340 |
+
# Initialize system state based on task
|
| 341 |
+
if task_id == "task1":
|
| 342 |
+
initial_sys_state = _create_initial_state_task1()
|
| 343 |
+
elif task_id == "task2":
|
| 344 |
+
initial_sys_state = _create_initial_state_task2()
|
| 345 |
+
elif task_id == "task3":
|
| 346 |
+
initial_sys_state = _create_initial_state_task3()
|
| 347 |
+
else:
|
| 348 |
+
initial_sys_state = {}
|
|
|
|
|
|
|
|
|
|
| 349 |
|
| 350 |
episode_id = str(uuid.uuid4())
|
| 351 |
_EPISODES[episode_id] = {
|
| 352 |
"task_id": task_id,
|
|
|
|
| 353 |
"step_number": 0,
|
| 354 |
"max_steps": meta["max_steps"],
|
| 355 |
"done": False,
|
| 356 |
"total_reward": 0.0,
|
| 357 |
"action_history": [],
|
| 358 |
"final_score": None,
|
| 359 |
+
"system_state": initial_sys_state,
|
| 360 |
}
|
| 361 |
|
| 362 |
+
system_state = _build_system_state(task_id, _EPISODES[episode_id])
|
| 363 |
+
|
| 364 |
return Observation(
|
| 365 |
task_id=task_id,
|
| 366 |
task_description=meta["description"],
|
| 367 |
episode_id=episode_id,
|
| 368 |
+
system_state=system_state,
|
| 369 |
thread_history=[],
|
| 370 |
available_actions=meta["available_actions"],
|
| 371 |
step_number=0,
|
| 372 |
max_steps=meta["max_steps"],
|
| 373 |
+
hint="Start by diagnosing the system state with basic commands.",
|
| 374 |
)
|
| 375 |
|
| 376 |
|
| 377 |
def step(episode_id: str, action: Action) -> StepResult:
|
| 378 |
+
"""Advance the episode by one step."""
|
| 379 |
+
ep = _EPISODES.get(episode_id)
|
| 380 |
+
if ep is None:
|
| 381 |
+
raise KeyError(f"Episode {episode_id} not found")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 382 |
|
| 383 |
if ep["done"]:
|
| 384 |
raise ValueError(f"Episode {episode_id} is already done.")
|
| 385 |
|
| 386 |
+
task_id = ep["task_id"]
|
| 387 |
meta = TASK_META[task_id]
|
|
|
|
| 388 |
|
| 389 |
ep["step_number"] += 1
|
| 390 |
ep["action_history"].append(action.model_dump())
|
| 391 |
|
| 392 |
+
# Execute action
|
| 393 |
+
if action.action_type == "bash_cmd":
|
| 394 |
+
cmd_output = _simulate_bash_cmd(action.command or "", task_id, ep)
|
| 395 |
+
ep["action_history"][-1]["output"] = cmd_output
|
| 396 |
+
elif action.action_type == "file_edit":
|
| 397 |
+
edit_result = _simulate_file_edit(action.file_path or "", action.file_content or "", ep)
|
| 398 |
+
ep["action_history"][-1]["result"] = edit_result
|
| 399 |
+
|
| 400 |
# Determine if done
|
| 401 |
done = False
|
| 402 |
if action.action_type == "submit":
|
|
|
|
| 404 |
elif ep["step_number"] >= ep["max_steps"]:
|
| 405 |
done = True
|
| 406 |
|
| 407 |
+
# Calculate reward
|
| 408 |
+
step_reward, explanation = _calculate_step_reward(task_id, action, ep)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 409 |
|
| 410 |
+
# Apply grader bonus when done
|
|
|
|
| 411 |
if done:
|
| 412 |
+
final_score, breakdown, grader_feedback = grade_task(task_id, ep)
|
|
|
|
|
|
|
| 413 |
ep["final_score"] = final_score
|
| 414 |
+
bonus = final_score * 0.5
|
| 415 |
+
step_reward += bonus
|
| 416 |
+
explanation += f" | Grader score: {final_score:.3f} (+{bonus:.3f} bonus)"
|
| 417 |
+
else:
|
| 418 |
+
final_score = None
|
| 419 |
|
| 420 |
+
ep["total_reward"] = round(ep["total_reward"] + step_reward, 4)
|
| 421 |
ep["done"] = done
|
| 422 |
|
| 423 |
+
# Build observation
|
| 424 |
+
system_state = _build_system_state(task_id, ep)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 425 |
thread_history = [
|
| 426 |
+
{"role": "agent", "content": str(a)} for a in ep["action_history"]
|
|
|
|
| 427 |
]
|
| 428 |
|
| 429 |
obs = Observation(
|
| 430 |
task_id=task_id,
|
| 431 |
task_description=meta["description"],
|
| 432 |
episode_id=episode_id,
|
| 433 |
+
system_state=system_state,
|
| 434 |
thread_history=thread_history,
|
| 435 |
available_actions=meta["available_actions"] if not done else [],
|
| 436 |
step_number=ep["step_number"],
|
| 437 |
max_steps=ep["max_steps"],
|
| 438 |
+
hint=None if done else "Continue diagnosing and fixing the issue.",
|
| 439 |
)
|
| 440 |
|
| 441 |
reward = Reward(
|
| 442 |
+
step_reward=round(step_reward, 4),
|
| 443 |
total_reward=ep["total_reward"],
|
| 444 |
explanation=explanation,
|
| 445 |
)
|
| 446 |
|
| 447 |
+
info = {"step": ep["step_number"]}
|
|
|
|
|
|
|
|
|
|
| 448 |
if done:
|
| 449 |
info["final_score"] = final_score
|
| 450 |
|
| 451 |
return StepResult(observation=obs, reward=reward, done=done, info=info)
|
| 452 |
|
| 453 |
|
| 454 |
+
def get_state(episode_id: str) -> State:
|
| 455 |
"""Return the current state of an episode."""
|
| 456 |
+
ep = _EPISODES.get(episode_id)
|
| 457 |
+
if ep is None:
|
| 458 |
+
raise KeyError(f"Episode {episode_id} not found")
|
| 459 |
+
|
| 460 |
return State(
|
| 461 |
task_id=ep["task_id"],
|
| 462 |
episode_id=episode_id,
|
|
|
|
| 469 |
)
|
| 470 |
|
| 471 |
|
| 472 |
+
def grade(episode_id: str) -> Tuple[float, Dict[str, float], str]:
|
| 473 |
+
"""Grade a finished episode."""
|
| 474 |
+
ep = _EPISODES.get(episode_id)
|
| 475 |
+
if ep is None:
|
| 476 |
+
raise KeyError(f"Episode {episode_id} not found")
|
| 477 |
|
| 478 |
+
if not ep.get("done"):
|
| 479 |
+
raise ValueError(f"Episode {episode_id} is not done yet")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 480 |
|
| 481 |
+
task_id = ep["task_id"]
|
| 482 |
+
score, breakdown, feedback = grade_task(task_id, ep)
|
|
|
|
| 483 |
ep["final_score"] = score
|
| 484 |
|
| 485 |
+
return score, breakdown, feedback
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 486 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
graders.py
CHANGED
|
@@ -1,397 +1,191 @@
|
|
| 1 |
-
|
| 2 |
"""
|
| 3 |
-
|
| 4 |
|
| 5 |
-
|
| 6 |
-
partial scores. No external calls are made — scoring is purely rule-based
|
| 7 |
-
so results are fully reproducible.
|
| 8 |
"""
|
| 9 |
-
from
|
| 10 |
-
|
| 11 |
-
import re
|
| 12 |
-
from typing import Any, Dict, List, Tuple
|
| 13 |
-
|
| 14 |
-
from data import TASK_META
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
# ---------------------------------------------------------------------------
|
| 18 |
-
# Helpers
|
| 19 |
-
# ---------------------------------------------------------------------------
|
| 20 |
|
| 21 |
-
def _normalise(text: str) -> str:
|
| 22 |
-
"""Lower-case, strip punctuation, collapse whitespace."""
|
| 23 |
-
text = text.lower()
|
| 24 |
-
text = re.sub(r"[^\w\s]", " ", text)
|
| 25 |
-
return re.sub(r"\s+", " ", text).strip()
|
| 26 |
|
| 27 |
-
|
| 28 |
-
def _contains_keyword(text: str, keyword: str) -> bool:
|
| 29 |
-
"""Return True if keyword (normalised) appears anywhere in text."""
|
| 30 |
-
return _normalise(keyword) in _normalise(text)
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
# ---------------------------------------------------------------------------
|
| 34 |
-
# Task 1 — Ticket Classification
|
| 35 |
-
# ---------------------------------------------------------------------------
|
| 36 |
-
|
| 37 |
-
def grade_task1(
|
| 38 |
-
ticket_index: int,
|
| 39 |
-
action_history: List[Dict[str, Any]],
|
| 40 |
-
) -> Tuple[float, Dict[str, float], str]:
|
| 41 |
"""
|
| 42 |
-
Grade a
|
| 43 |
-
|
| 44 |
-
Scoring:
|
| 45 |
-
category_correct : 0.50
|
| 46 |
-
priority_correct : 0.40
|
| 47 |
-
acted_efficiently : 0.10 (submitted within max_steps without wasted loops)
|
| 48 |
-
|
| 49 |
Returns (score, breakdown, feedback)
|
| 50 |
"""
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
# Find the last classify action the agent submitted
|
| 58 |
-
last_classify = None
|
| 59 |
-
for act in reversed(action_history):
|
| 60 |
-
if act.get("action_type") == "classify":
|
| 61 |
-
last_classify = act
|
| 62 |
-
break
|
| 63 |
-
|
| 64 |
-
if last_classify is not None:
|
| 65 |
-
if last_classify.get("category", "").lower() == gt["category"]:
|
| 66 |
-
category_score = 0.50
|
| 67 |
-
elif _normalise(last_classify.get("category", "")) in _normalise(gt["category"]):
|
| 68 |
-
category_score = 0.25 # partial credit for substring match
|
| 69 |
-
|
| 70 |
-
if last_classify.get("priority", "").lower() == gt["priority"]:
|
| 71 |
-
priority_score = 0.40
|
| 72 |
-
elif _is_adjacent_priority(
|
| 73 |
-
last_classify.get("priority", "").lower(), gt["priority"]
|
| 74 |
-
):
|
| 75 |
-
priority_score = 0.20 # partial credit for one level off
|
| 76 |
-
|
| 77 |
-
# Efficiency bonus: agent didn't fill max steps with redundant classify attempts
|
| 78 |
-
classify_count = sum(
|
| 79 |
-
1 for a in action_history if a.get("action_type") == "classify"
|
| 80 |
-
)
|
| 81 |
-
efficiency_score = 0.10 if classify_count <= 2 else 0.0
|
| 82 |
-
|
| 83 |
-
total = category_score + priority_score + efficiency_score
|
| 84 |
-
breakdown = {
|
| 85 |
-
"category_correct": category_score,
|
| 86 |
-
"priority_correct": priority_score,
|
| 87 |
-
"efficiency": efficiency_score,
|
| 88 |
-
}
|
| 89 |
-
feedback = _task1_feedback(gt, last_classify, breakdown)
|
| 90 |
-
return round(total, 4), breakdown, feedback
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
def _is_adjacent_priority(pred: str, gold: str) -> bool:
|
| 94 |
-
"""One priority level off gets partial credit."""
|
| 95 |
-
levels = ["low", "medium", "high", "critical"]
|
| 96 |
-
if pred not in levels or gold not in levels:
|
| 97 |
-
return False
|
| 98 |
-
return abs(levels.index(pred) - levels.index(gold)) == 1
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
def _task1_feedback(gt, last_classify, breakdown) -> str:
|
| 102 |
-
parts = []
|
| 103 |
-
if last_classify is None:
|
| 104 |
-
return "No classify action found in episode history."
|
| 105 |
-
if breakdown["category_correct"] == 0.50:
|
| 106 |
-
parts.append(f"Category '{gt['category']}' correctly identified.")
|
| 107 |
-
elif breakdown["category_correct"] > 0:
|
| 108 |
-
parts.append(f"Category partially correct (got substring match).")
|
| 109 |
-
else:
|
| 110 |
-
parts.append(
|
| 111 |
-
f"Category incorrect: expected '{gt['category']}', "
|
| 112 |
-
f"got '{last_classify.get('category')}'."
|
| 113 |
-
)
|
| 114 |
-
if breakdown["priority_correct"] == 0.40:
|
| 115 |
-
parts.append(f"Priority '{gt['priority']}' correctly identified.")
|
| 116 |
-
elif breakdown["priority_correct"] > 0:
|
| 117 |
-
parts.append("Priority off by one level (partial credit).")
|
| 118 |
else:
|
| 119 |
-
|
| 120 |
-
f"Priority incorrect: expected '{gt['priority']}', "
|
| 121 |
-
f"got '{last_classify.get('priority')}'."
|
| 122 |
-
)
|
| 123 |
-
return " ".join(parts)
|
| 124 |
|
| 125 |
|
| 126 |
-
|
| 127 |
-
# Task 2 — Information Extraction
|
| 128 |
-
# ---------------------------------------------------------------------------
|
| 129 |
-
|
| 130 |
-
def grade_task2(
|
| 131 |
-
ticket_index: int,
|
| 132 |
-
action_history: List[Dict[str, Any]],
|
| 133 |
-
) -> Tuple[float, Dict[str, float], str]:
|
| 134 |
"""
|
| 135 |
-
Grade
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
Returns (score, breakdown, feedback)
|
| 143 |
"""
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
required_actions: List[str] = gt["required_actions"]
|
| 148 |
-
|
| 149 |
-
# Find the last extract action
|
| 150 |
-
last_extract = None
|
| 151 |
-
for act in reversed(action_history):
|
| 152 |
-
if act.get("action_type") == "extract":
|
| 153 |
-
last_extract = act
|
| 154 |
-
break
|
| 155 |
-
|
| 156 |
-
if last_extract is None:
|
| 157 |
-
return 0.0, {"entity_coverage": 0.0, "action_coverage": 0.0, "no_hallucination": 0.0}, \
|
| 158 |
-
"No extract action found in episode history."
|
| 159 |
-
|
| 160 |
-
extracted: Dict[str, Any] = last_extract.get("extracted_entities") or {}
|
| 161 |
-
submitted_actions: List[str] = last_extract.get("required_actions") or []
|
| 162 |
-
|
| 163 |
-
# --- Entity scoring ---
|
| 164 |
-
entity_hits = 0
|
| 165 |
-
entity_notes = []
|
| 166 |
-
for key, expected_val in required_entities.items():
|
| 167 |
-
pred_val = extracted.get(key)
|
| 168 |
-
if pred_val is None:
|
| 169 |
-
entity_notes.append(f"Missing: {key}")
|
| 170 |
-
continue
|
| 171 |
-
if isinstance(expected_val, list):
|
| 172 |
-
# For list values, check set overlap (case-insensitive)
|
| 173 |
-
pred_set = {_normalise(str(v)) for v in (pred_val if isinstance(pred_val, list) else [pred_val])}
|
| 174 |
-
gold_set = {_normalise(str(v)) for v in expected_val}
|
| 175 |
-
overlap = len(pred_set & gold_set) / len(gold_set)
|
| 176 |
-
entity_hits += overlap
|
| 177 |
-
if overlap < 1.0:
|
| 178 |
-
entity_notes.append(f"Partial match for '{key}': {overlap:.0%}")
|
| 179 |
-
else:
|
| 180 |
-
if _normalise(str(pred_val)) == _normalise(str(expected_val)):
|
| 181 |
-
entity_hits += 1
|
| 182 |
-
elif _normalise(str(expected_val)) in _normalise(str(pred_val)):
|
| 183 |
-
entity_hits += 0.5
|
| 184 |
-
entity_notes.append(f"Partial match for '{key}'")
|
| 185 |
-
else:
|
| 186 |
-
entity_notes.append(
|
| 187 |
-
f"Wrong value for '{key}': expected '{expected_val}', got '{pred_val}'"
|
| 188 |
-
)
|
| 189 |
-
|
| 190 |
-
entity_coverage = entity_hits / max(len(required_entities), 1)
|
| 191 |
-
|
| 192 |
-
# --- Action scoring ---
|
| 193 |
-
gold_action_set = {a.lower() for a in required_actions}
|
| 194 |
-
pred_action_set = {a.lower() for a in submitted_actions}
|
| 195 |
-
action_hits = len(gold_action_set & pred_action_set)
|
| 196 |
-
# Partial credit: normalised keyword match
|
| 197 |
-
for gold_act in gold_action_set:
|
| 198 |
-
if gold_act not in pred_action_set:
|
| 199 |
-
for pred_act in pred_action_set:
|
| 200 |
-
if _normalise(gold_act) in _normalise(pred_act) or \
|
| 201 |
-
_normalise(pred_act) in _normalise(gold_act):
|
| 202 |
-
action_hits += 0.5
|
| 203 |
-
break
|
| 204 |
-
action_coverage = min(action_hits / max(len(required_actions), 1), 1.0)
|
| 205 |
-
|
| 206 |
-
# --- Hallucination penalty ---
|
| 207 |
-
# Extra keys in extracted that don't exist in required_entities
|
| 208 |
-
extra_keys = set(extracted.keys()) - set(required_entities.keys())
|
| 209 |
-
hallucination_score = 0.10 if len(extra_keys) <= 2 else 0.0
|
| 210 |
-
|
| 211 |
-
total = (
|
| 212 |
-
entity_coverage * 0.60
|
| 213 |
-
+ action_coverage * 0.30
|
| 214 |
-
+ hallucination_score
|
| 215 |
-
)
|
| 216 |
-
|
| 217 |
breakdown = {
|
| 218 |
-
"
|
| 219 |
-
"
|
| 220 |
-
"
|
| 221 |
}
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 232 |
|
| 233 |
-
# ---------------------------------------------------------------------------
|
| 234 |
-
# Task 3 — Resolution Generation
|
| 235 |
-
# ---------------------------------------------------------------------------
|
| 236 |
|
| 237 |
-
def
|
| 238 |
-
ticket_index: int,
|
| 239 |
-
action_history: List[Dict[str, Any]],
|
| 240 |
-
) -> Tuple[float, Dict[str, float], str]:
|
| 241 |
"""
|
| 242 |
-
Grade
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
no_empty_steps : 0.05 (no empty or single-word steps)
|
| 250 |
-
|
| 251 |
-
Returns (score, breakdown, feedback)
|
| 252 |
"""
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 265 |
break
|
| 266 |
-
|
| 267 |
-
if
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
"
|
| 272 |
-
|
| 273 |
-
"no_empty_steps": 0.0,
|
| 274 |
-
}, "No respond/resolve action found in episode history."
|
| 275 |
-
|
| 276 |
-
response_text: str = last_response_act.get("response_text") or ""
|
| 277 |
-
resolution_steps: List[str] = last_response_act.get("resolution_steps") or []
|
| 278 |
-
|
| 279 |
-
# --- Keyword coverage ---
|
| 280 |
-
kw_hits = sum(
|
| 281 |
-
1 for kw in required_keywords if _contains_keyword(response_text, kw)
|
| 282 |
-
)
|
| 283 |
-
keyword_coverage = kw_hits / max(len(required_keywords), 1)
|
| 284 |
-
|
| 285 |
-
# --- Step coverage ---
|
| 286 |
-
step_hits = 0.0
|
| 287 |
-
for gold_step in required_steps:
|
| 288 |
-
# A step is "covered" if any resolution_step string contains all
|
| 289 |
-
# main words from the gold step label (split on underscores)
|
| 290 |
-
main_words = gold_step.replace("_", " ").split()
|
| 291 |
-
for pred_step in resolution_steps:
|
| 292 |
-
matches = sum(1 for w in main_words if _contains_keyword(pred_step, w))
|
| 293 |
-
if matches == len(main_words):
|
| 294 |
-
step_hits += 1
|
| 295 |
break
|
| 296 |
-
|
| 297 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 298 |
break
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
# --- Length adequacy ---
|
| 305 |
-
word_count = len(response_text.split())
|
| 306 |
-
length_score = 0.10 if word_count >= min_length else round(
|
| 307 |
-
0.10 * word_count / min_length, 4
|
| 308 |
-
)
|
| 309 |
-
|
| 310 |
-
# --- Step quality ---
|
| 311 |
-
empty_step_count = sum(
|
| 312 |
-
1 for s in resolution_steps if len(s.strip().split()) <= 1
|
| 313 |
-
)
|
| 314 |
-
no_empty_steps_score = 0.05 if empty_step_count == 0 else 0.0
|
| 315 |
|
| 316 |
-
total = (
|
| 317 |
-
keyword_coverage * 0.30
|
| 318 |
-
+ step_coverage * 0.30
|
| 319 |
-
+ tone_score
|
| 320 |
-
+ length_score
|
| 321 |
-
+ no_empty_steps_score
|
| 322 |
-
)
|
| 323 |
|
| 324 |
-
|
| 325 |
-
"keyword_coverage": round(keyword_coverage * 0.30, 4),
|
| 326 |
-
"step_coverage": round(step_coverage * 0.30, 4),
|
| 327 |
-
"tone_compliance": round(tone_score, 4),
|
| 328 |
-
"length_adequate": round(length_score, 4),
|
| 329 |
-
"no_empty_steps": no_empty_steps_score,
|
| 330 |
-
}
|
| 331 |
-
feedback = (
|
| 332 |
-
f"Keywords: {kw_hits}/{len(required_keywords)} found. "
|
| 333 |
-
f"Steps: {step_hits:.1f}/{len(required_steps)} covered. "
|
| 334 |
-
f"Tone score: {tone_score:.2f}/0.25. "
|
| 335 |
-
f"Word count: {word_count} (min {min_length})."
|
| 336 |
-
)
|
| 337 |
-
|
| 338 |
-
return round(total, 4), breakdown, feedback
|
| 339 |
-
|
| 340 |
-
|
| 341 |
-
_APOLOGY_WORDS = [
|
| 342 |
-
"sorry", "apologise", "apologize", "apologies", "regret",
|
| 343 |
-
"inconvenience", "unfortunately",
|
| 344 |
-
]
|
| 345 |
-
_URGENCY_WORDS = [
|
| 346 |
-
"urgent", "urgently", "immediately", "priority", "asap",
|
| 347 |
-
"right away", "as soon as possible",
|
| 348 |
-
]
|
| 349 |
-
_TIMELINE_WORDS = [
|
| 350 |
-
"within", "hour", "minute", "today", "by", "shortly",
|
| 351 |
-
"soon", "moment", "immediately", "right away",
|
| 352 |
-
]
|
| 353 |
-
|
| 354 |
-
|
| 355 |
-
def _check_tone(text: str, tone: Dict[str, bool]) -> float:
|
| 356 |
-
"""Return tone compliance score (max 0.25)."""
|
| 357 |
-
criteria = [
|
| 358 |
-
("must_apologize", _APOLOGY_WORDS),
|
| 359 |
-
("must_acknowledge_urgency", _URGENCY_WORDS),
|
| 360 |
-
("must_provide_timeline", _TIMELINE_WORDS),
|
| 361 |
-
]
|
| 362 |
-
required = [k for k, _ in criteria if tone.get(k, False)]
|
| 363 |
-
if not required:
|
| 364 |
-
return 0.25 # nothing required → full tone score
|
| 365 |
-
|
| 366 |
-
per_criterion = 0.25 / len(required)
|
| 367 |
-
score = 0.0
|
| 368 |
-
for key, words in criteria:
|
| 369 |
-
if not tone.get(key, False):
|
| 370 |
-
continue
|
| 371 |
-
if any(_contains_keyword(text, w) for w in words):
|
| 372 |
-
score += per_criterion
|
| 373 |
-
return round(score, 4)
|
| 374 |
-
|
| 375 |
-
|
| 376 |
-
# ---------------------------------------------------------------------------
|
| 377 |
-
# Public grader dispatcher
|
| 378 |
-
# ---------------------------------------------------------------------------
|
| 379 |
-
|
| 380 |
-
def grade_episode(
|
| 381 |
-
task_id: str,
|
| 382 |
-
ticket_index: int,
|
| 383 |
-
action_history: List[Dict[str, Any]],
|
| 384 |
-
) -> Tuple[float, Dict[str, float], str]:
|
| 385 |
"""
|
| 386 |
-
|
| 387 |
-
|
| 388 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 389 |
"""
|
| 390 |
-
|
| 391 |
-
|
| 392 |
-
|
| 393 |
-
|
| 394 |
-
|
| 395 |
-
|
| 396 |
-
|
| 397 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
"""
|
| 2 |
+
Graders for DevOpsEnv tasks.
|
| 3 |
|
| 4 |
+
Deterministic scoring based on system state changes and action validity.
|
|
|
|
|
|
|
| 5 |
"""
|
| 6 |
+
from typing import Any, Dict, Tuple
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
+
def grade_task(task_id: str, episode_state: Dict[str, Any]) -> Tuple[float, Dict[str, float], str]:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
"""
|
| 11 |
+
Grade a completed task episode.
|
| 12 |
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
Returns (score, breakdown, feedback)
|
| 14 |
"""
|
| 15 |
+
if task_id == "task1":
|
| 16 |
+
return grade_task1(episode_state)
|
| 17 |
+
elif task_id == "task2":
|
| 18 |
+
return grade_task2(episode_state)
|
| 19 |
+
elif task_id == "task3":
|
| 20 |
+
return grade_task3(episode_state)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
else:
|
| 22 |
+
return 0.0, {}, "Unknown task"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
|
| 25 |
+
def grade_task1(episode_state: Dict[str, Any]) -> Tuple[float, Dict[str, float], str]:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
"""
|
| 27 |
+
Grade Task 1: Restart Nginx.
|
| 28 |
+
|
| 29 |
+
Success criteria:
|
| 30 |
+
- nginx service is running (30%)
|
| 31 |
+
- nginx config is valid (30%)
|
| 32 |
+
- HTTP 200 response on port 80 (40%)
|
|
|
|
|
|
|
| 33 |
"""
|
| 34 |
+
state_dict = episode_state.get("system_state", {})
|
| 35 |
+
action_history = episode_state.get("action_history", [])
|
| 36 |
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
breakdown = {
|
| 38 |
+
"nginx_running": 0.0,
|
| 39 |
+
"config_valid": 0.0,
|
| 40 |
+
"http_200": 0.0,
|
| 41 |
}
|
| 42 |
+
|
| 43 |
+
# Check if nginx is running
|
| 44 |
+
service_status = state_dict.get("service_status", {})
|
| 45 |
+
if service_status.get("nginx") == "active":
|
| 46 |
+
breakdown["nginx_running"] = 0.3
|
| 47 |
+
|
| 48 |
+
# Check if config validation was attempted and passed
|
| 49 |
+
config_valid = False
|
| 50 |
+
for action in action_history:
|
| 51 |
+
output = action.get("output", "")
|
| 52 |
+
if output and ("syntax is ok" in str(output).lower() or "test is successful" in str(output).lower()):
|
| 53 |
+
config_valid = True
|
| 54 |
+
breakdown["config_valid"] = 0.3
|
| 55 |
+
break
|
| 56 |
+
|
| 57 |
+
# Check if HTTP 200 response was achieved
|
| 58 |
+
http_ports = state_dict.get("http_ports_open", [])
|
| 59 |
+
if 80 in http_ports:
|
| 60 |
+
# Verify http 200 response was confirmed
|
| 61 |
+
for action in action_history:
|
| 62 |
+
output = action.get("output", "")
|
| 63 |
+
cmd = action.get("command", "")
|
| 64 |
+
if output and cmd and "OK" in str(output) and "curl" in str(cmd).lower():
|
| 65 |
+
breakdown["http_200"] = 0.4
|
| 66 |
+
break
|
| 67 |
+
|
| 68 |
+
score = sum(breakdown.values())
|
| 69 |
+
feedback = f"Task 1 Grading: nginx_running={breakdown['nginx_running']:.1f}, config_valid={breakdown['config_valid']:.1f}, http_200={breakdown['http_200']:.1f}"
|
| 70 |
+
|
| 71 |
+
return min(score, 1.0), breakdown, feedback
|
| 72 |
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
+
def grade_task2(episode_state: Dict[str, Any]) -> Tuple[float, Dict[str, float], str]:
|
|
|
|
|
|
|
|
|
|
| 75 |
"""
|
| 76 |
+
Grade Task 2: Fix Docker configuration.
|
| 77 |
+
|
| 78 |
+
Success criteria:
|
| 79 |
+
- docker-compose.yml was edited (25%)
|
| 80 |
+
- docker-compose up -d was successful (25%)
|
| 81 |
+
- Container is running (25%)
|
| 82 |
+
- Service accessible on correct port (25%)
|
|
|
|
|
|
|
|
|
|
| 83 |
"""
|
| 84 |
+
state_dict = episode_state.get("system_state", {})
|
| 85 |
+
action_history = episode_state.get("action_history", [])
|
| 86 |
+
files = state_dict.get("files", {})
|
| 87 |
+
|
| 88 |
+
breakdown = {
|
| 89 |
+
"file_edited": 0.0,
|
| 90 |
+
"compose_ran": 0.0,
|
| 91 |
+
"container_running": 0.0,
|
| 92 |
+
"port_accessible": 0.0,
|
| 93 |
+
}
|
| 94 |
+
|
| 95 |
+
# Check if docker-compose.yml was edited correctly
|
| 96 |
+
compose_file = "/srv/docker-compose.yml"
|
| 97 |
+
if compose_file in files:
|
| 98 |
+
content = files[compose_file]
|
| 99 |
+
if content and "3000:3000" in str(content):
|
| 100 |
+
breakdown["file_edited"] = 0.25
|
| 101 |
+
|
| 102 |
+
# Check if docker-compose up -d was run
|
| 103 |
+
for action in action_history:
|
| 104 |
+
cmd = action.get("command")
|
| 105 |
+
if cmd and "docker-compose up -d" in str(cmd):
|
| 106 |
+
output = action.get("output", "")
|
| 107 |
+
if output and ("done" in str(output).lower() or "created" in str(output).lower()):
|
| 108 |
+
breakdown["compose_ran"] = 0.25
|
| 109 |
break
|
| 110 |
+
|
| 111 |
+
# Check if container is running
|
| 112 |
+
containers = state_dict.get("docker_containers", [])
|
| 113 |
+
if containers:
|
| 114 |
+
for container in containers:
|
| 115 |
+
if container.get("status") == "running" and "mockapi" in str(container.get("name", "")):
|
| 116 |
+
breakdown["container_running"] = 0.25
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
break
|
| 118 |
+
|
| 119 |
+
# Check if port is correctly mapped
|
| 120 |
+
if containers:
|
| 121 |
+
for container in containers:
|
| 122 |
+
if "3000:3000" in str(container.get("ports", "")):
|
| 123 |
+
breakdown["port_accessible"] = 0.25
|
| 124 |
break
|
| 125 |
+
|
| 126 |
+
score = sum(breakdown.values())
|
| 127 |
+
feedback = f"Task 2 Grading: file_edited={breakdown['file_edited']:.2f}, compose_ran={breakdown['compose_ran']:.2f}, container_running={breakdown['container_running']:.2f}, port_accessible={breakdown['port_accessible']:.2f}"
|
| 128 |
+
|
| 129 |
+
return min(score, 1.0), breakdown, feedback
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
+
def grade_task3(episode_state: Dict[str, Any]) -> Tuple[float, Dict[str, float], str]:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
"""
|
| 134 |
+
Grade Task 3: Fix memory leak.
|
| 135 |
+
|
| 136 |
+
Success criteria:
|
| 137 |
+
- Process was killed (25%)
|
| 138 |
+
- Code was fixed (removing the leak) (25%)
|
| 139 |
+
- Service was restarted (25%)
|
| 140 |
+
- Memory usage decreased (25%)
|
| 141 |
"""
|
| 142 |
+
state_dict = episode_state.get("system_state", {})
|
| 143 |
+
action_history = episode_state.get("action_history", [])
|
| 144 |
+
files = state_dict.get("files", {})
|
| 145 |
+
processes = state_dict.get("running_processes", [])
|
| 146 |
+
|
| 147 |
+
breakdown = {
|
| 148 |
+
"process_killed": 0.0,
|
| 149 |
+
"code_fixed": 0.0,
|
| 150 |
+
"service_restarted": 0.0,
|
| 151 |
+
"memory_reduced": 0.0,
|
| 152 |
+
}
|
| 153 |
+
|
| 154 |
+
# Check if python process was killed
|
| 155 |
+
has_python_leak = False
|
| 156 |
+
if processes:
|
| 157 |
+
has_python_leak = any(p.get("name") == "python3" and p.get("rss_mb", 512) > 1024 for p in processes)
|
| 158 |
+
if not has_python_leak:
|
| 159 |
+
# Process was killed
|
| 160 |
+
breakdown["process_killed"] = 0.25
|
| 161 |
+
|
| 162 |
+
# Check if code was fixed (removed the memory leak)
|
| 163 |
+
app_file = "/opt/mockapi/app.py"
|
| 164 |
+
if app_file in files:
|
| 165 |
+
content = files[app_file]
|
| 166 |
+
# Memory leak is the unbounded list append - check if it is fixed
|
| 167 |
+
if content and ("request_cache.append" not in str(content) or "request_cache = []" not in str(content)):
|
| 168 |
+
# If it has been removed or replaced with something better
|
| 169 |
+
if "request_cache" not in str(content) or "# " in str(content):
|
| 170 |
+
breakdown["code_fixed"] = 0.25
|
| 171 |
+
|
| 172 |
+
# Check if service was restarted
|
| 173 |
+
service_status = state_dict.get("service_status", {})
|
| 174 |
+
if service_status.get("mockapi") == "active":
|
| 175 |
+
# And there is a newer process
|
| 176 |
+
for action in action_history:
|
| 177 |
+
cmd = action.get("command", "")
|
| 178 |
+
if cmd and "python3" in str(cmd) and ("start" in str(cmd) or "&" in str(cmd)):
|
| 179 |
+
breakdown["service_restarted"] = 0.25
|
| 180 |
+
break
|
| 181 |
+
|
| 182 |
+
# Check if memory usage decreased
|
| 183 |
+
initial_memory = 2048
|
| 184 |
+
current_memory = state_dict.get("memory_usage_mb", 2048)
|
| 185 |
+
if current_memory < initial_memory * 0.75: # At least 25% improvement
|
| 186 |
+
breakdown["memory_reduced"] = 0.25
|
| 187 |
+
|
| 188 |
+
score = sum(breakdown.values())
|
| 189 |
+
feedback = f"Task 3 Grading: process_killed={breakdown['process_killed']:.2f}, code_fixed={breakdown['code_fixed']:.2f}, service_restarted={breakdown['service_restarted']:.2f}, memory_reduced={breakdown['memory_reduced']:.2f}"
|
| 190 |
+
|
| 191 |
+
return min(score, 1.0), breakdown, feedback
|
inference.py
ADDED
|
@@ -0,0 +1,352 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Baseline LLM inference agent for DevOpsEnv.
|
| 3 |
+
|
| 4 |
+
This script reads an OpenEnv environment's state() and uses an LLM to generate
|
| 5 |
+
actions that solve the DevOps tasks.
|
| 6 |
+
|
| 7 |
+
Usage:
|
| 8 |
+
python inference.py --task task1 --model gpt-4 --hf-token <token>
|
| 9 |
+
"""
|
| 10 |
+
import os
|
| 11 |
+
import sys
|
| 12 |
+
import json
|
| 13 |
+
import argparse
|
| 14 |
+
import time
|
| 15 |
+
from pathlib import Path
|
| 16 |
+
from typing import Optional
|
| 17 |
+
|
| 18 |
+
import requests
|
| 19 |
+
from google import genai
|
| 20 |
+
from openai import OpenAI
|
| 21 |
+
|
| 22 |
+
# Load .env values from current folder (if present) before reading config.
|
| 23 |
+
def _load_dotenv_from_workspace() -> None:
|
| 24 |
+
"""Load KEY=VALUE pairs from .env into os.environ without overriding existing vars."""
|
| 25 |
+
dotenv_path = Path(__file__).resolve().parent / ".env"
|
| 26 |
+
if not dotenv_path.exists():
|
| 27 |
+
return
|
| 28 |
+
|
| 29 |
+
for raw_line in dotenv_path.read_text(encoding="utf-8").splitlines():
|
| 30 |
+
line = raw_line.strip()
|
| 31 |
+
if not line or line.startswith("#"):
|
| 32 |
+
continue
|
| 33 |
+
if line.startswith("export "):
|
| 34 |
+
line = line[7:].strip()
|
| 35 |
+
if "=" not in line:
|
| 36 |
+
continue
|
| 37 |
+
|
| 38 |
+
key, value = line.split("=", 1)
|
| 39 |
+
key = key.strip()
|
| 40 |
+
value = value.strip()
|
| 41 |
+
if not key:
|
| 42 |
+
continue
|
| 43 |
+
|
| 44 |
+
# Remove surrounding quotes if present.
|
| 45 |
+
if (value.startswith('"') and value.endswith('"')) or (
|
| 46 |
+
value.startswith("'") and value.endswith("'")
|
| 47 |
+
):
|
| 48 |
+
value = value[1:-1]
|
| 49 |
+
|
| 50 |
+
os.environ.setdefault(key, value)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
_load_dotenv_from_workspace()
|
| 54 |
+
|
| 55 |
+
# Read config from environment/.env
|
| 56 |
+
API_BASE_URL = os.environ.get("API_BASE_URL", "http://localhost:7860")
|
| 57 |
+
MODEL_NAME = os.environ.get("MODEL_NAME", "gpt-4")
|
| 58 |
+
HF_TOKEN = os.environ.get("HF_TOKEN", "")
|
| 59 |
+
OPENAI_BASE_URL = os.environ.get("OPENAI_BASE_URL", "")
|
| 60 |
+
GEMINI_DEFAULT_MODEL = os.environ.get("GEMINI_MODEL", "gemini-3-flash-preview")
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def _get_openai_client() -> OpenAI:
|
| 64 |
+
"""Create an OpenAI-compatible client for OpenAI-style chat completions."""
|
| 65 |
+
api_key = os.environ.get("OPENAI_API_KEY", "sk-test")
|
| 66 |
+
client_kwargs = {"api_key": api_key}
|
| 67 |
+
if OPENAI_BASE_URL:
|
| 68 |
+
client_kwargs["base_url"] = OPENAI_BASE_URL
|
| 69 |
+
return OpenAI(**client_kwargs)
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def _get_gemini_client() -> genai.Client:
|
| 73 |
+
"""Create a Gemini client using the official google-genai SDK."""
|
| 74 |
+
api_key = os.environ.get("GEMINI_API_KEY") or os.environ.get("OPENAI_API_KEY", "")
|
| 75 |
+
if not api_key:
|
| 76 |
+
raise ValueError("GEMINI_API_KEY is required for Gemini models")
|
| 77 |
+
return genai.Client(api_key=api_key)
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def _is_gemini_model(model: str) -> bool:
|
| 81 |
+
"""Detect whether the requested model should use the Gemini SDK path."""
|
| 82 |
+
m = (model or "").lower()
|
| 83 |
+
return "gemini" in m
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def _resolve_gemini_model(model: str) -> str:
|
| 87 |
+
"""Map shorthand Gemini model names to concrete model IDs."""
|
| 88 |
+
m = (model or "").strip()
|
| 89 |
+
if not m or m.lower() == "gemini":
|
| 90 |
+
return GEMINI_DEFAULT_MODEL
|
| 91 |
+
return m
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def _generate_action_text(
|
| 95 |
+
model: str,
|
| 96 |
+
system_prompt: str,
|
| 97 |
+
user_prompt: str,
|
| 98 |
+
openai_client: Optional[OpenAI],
|
| 99 |
+
gemini_client: Optional[genai.Client],
|
| 100 |
+
) -> str:
|
| 101 |
+
"""Generate model output text using Gemini SDK or OpenAI-compatible chat."""
|
| 102 |
+
if _is_gemini_model(model):
|
| 103 |
+
if gemini_client is None:
|
| 104 |
+
raise ValueError("Gemini client was not initialized")
|
| 105 |
+
gemini_model = _resolve_gemini_model(model)
|
| 106 |
+
combined_prompt = (
|
| 107 |
+
f"System instructions:\n{system_prompt}\n\n"
|
| 108 |
+
f"User request:\n{user_prompt}"
|
| 109 |
+
)
|
| 110 |
+
response = gemini_client.models.generate_content(
|
| 111 |
+
model=gemini_model,
|
| 112 |
+
contents=combined_prompt,
|
| 113 |
+
)
|
| 114 |
+
return response.text or ""
|
| 115 |
+
|
| 116 |
+
if openai_client is None:
|
| 117 |
+
raise ValueError("OpenAI client was not initialized")
|
| 118 |
+
|
| 119 |
+
response = openai_client.chat.completions.create(
|
| 120 |
+
model=model,
|
| 121 |
+
messages=[
|
| 122 |
+
{"role": "system", "content": system_prompt},
|
| 123 |
+
{"role": "user", "content": user_prompt},
|
| 124 |
+
],
|
| 125 |
+
temperature=0.3,
|
| 126 |
+
max_tokens=1000,
|
| 127 |
+
)
|
| 128 |
+
return response.choices[0].message.content or ""
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def send_request(method: str, endpoint: str, **kwargs):
|
| 132 |
+
"""Send HTTP request to the environment server."""
|
| 133 |
+
url = f"{API_BASE_URL}{endpoint}"
|
| 134 |
+
response = requests.request(method, url, timeout=10, **kwargs)
|
| 135 |
+
response.raise_for_status()
|
| 136 |
+
return response.json()
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def run_agent(task_id: str, max_steps: int = 20, model: Optional[str] = None) -> dict:
|
| 140 |
+
"""Run the agent on a specific task."""
|
| 141 |
+
model = model or MODEL_NAME
|
| 142 |
+
if _is_gemini_model(model):
|
| 143 |
+
model = _resolve_gemini_model(model)
|
| 144 |
+
openai_client: Optional[OpenAI] = None
|
| 145 |
+
gemini_client: Optional[genai.Client] = None
|
| 146 |
+
if _is_gemini_model(model):
|
| 147 |
+
gemini_client = _get_gemini_client()
|
| 148 |
+
else:
|
| 149 |
+
openai_client = _get_openai_client()
|
| 150 |
+
|
| 151 |
+
# Initialize episode
|
| 152 |
+
print(f"\n{'='*60}")
|
| 153 |
+
print(f"Starting task: {task_id}")
|
| 154 |
+
print(f"Model: {model}")
|
| 155 |
+
print(f"{'='*60}\n")
|
| 156 |
+
|
| 157 |
+
obs = send_request("POST", "/reset", json={"task_id": task_id})
|
| 158 |
+
episode_id = obs["episode_id"]
|
| 159 |
+
max_steps = obs["max_steps"]
|
| 160 |
+
|
| 161 |
+
print(f"Episode ID: {episode_id}")
|
| 162 |
+
print(f"Max Steps: {max_steps}")
|
| 163 |
+
print(f"\nTask: {obs['task_description']}\n")
|
| 164 |
+
|
| 165 |
+
step_count = 0
|
| 166 |
+
total_reward = 0.0
|
| 167 |
+
actions_taken = []
|
| 168 |
+
|
| 169 |
+
while step_count < max_steps:
|
| 170 |
+
step_count += 1
|
| 171 |
+
|
| 172 |
+
# Get current state
|
| 173 |
+
state = send_request("GET", f"/state?episode_id={episode_id}")
|
| 174 |
+
|
| 175 |
+
# Prepare prompt for LLM
|
| 176 |
+
system_prompt = """You are an expert Linux DevOps engineer/SRE.
|
| 177 |
+
Your job is to diagnose and fix broken systems using bash commands and file edits.
|
| 178 |
+
You are interacting with a simulated Linux environment.
|
| 179 |
+
|
| 180 |
+
Available actions:
|
| 181 |
+
1. bash_cmd: Execute a bash command
|
| 182 |
+
2. file_edit: Edit a file
|
| 183 |
+
3. submit: Submit when the task is complete
|
| 184 |
+
|
| 185 |
+
Respond in JSON format with this structure:
|
| 186 |
+
{
|
| 187 |
+
"action_type": "bash_cmd" | "file_edit" | "submit",
|
| 188 |
+
"command": "command to execute" (if bash_cmd),
|
| 189 |
+
"file_path": "/path/to/file" (if file_edit),
|
| 190 |
+
"file_content": "new file content" (if file_edit),
|
| 191 |
+
"summary": "why you're taking this action"
|
| 192 |
+
}
|
| 193 |
+
|
| 194 |
+
Be strategic:
|
| 195 |
+
- Start by diagnosing the system
|
| 196 |
+
- Use ps, systemctl, curl, etc. to understand issues
|
| 197 |
+
- Fix the root cause
|
| 198 |
+
- Submit when done
|
| 199 |
+
"""
|
| 200 |
+
|
| 201 |
+
user_prompt = f"""
|
| 202 |
+
Current system state:
|
| 203 |
+
- Task: {obs['task_description']}
|
| 204 |
+
- Step: {state['step_number']}/{state['max_steps']}
|
| 205 |
+
- Reward so far: {state['total_reward']:.3f}
|
| 206 |
+
|
| 207 |
+
System status:
|
| 208 |
+
{json.dumps(obs['system_state'], indent=2)}
|
| 209 |
+
|
| 210 |
+
Previous actions: {len(state['history'])} taken so far
|
| 211 |
+
|
| 212 |
+
History of commands:
|
| 213 |
+
{json.dumps(state['history'][-3:], indent=2) if state['history'] else 'None yet'}
|
| 214 |
+
|
| 215 |
+
What should I do next? Think step-by-step about what the issue is and how to fix it.
|
| 216 |
+
"""
|
| 217 |
+
|
| 218 |
+
try:
|
| 219 |
+
# Call LLM (Gemini SDK or OpenAI-compatible chat)
|
| 220 |
+
response_text = _generate_action_text(
|
| 221 |
+
model=model,
|
| 222 |
+
system_prompt=system_prompt,
|
| 223 |
+
user_prompt=user_prompt,
|
| 224 |
+
openai_client=openai_client,
|
| 225 |
+
gemini_client=gemini_client,
|
| 226 |
+
)
|
| 227 |
+
try:
|
| 228 |
+
# Try to extract JSON from response
|
| 229 |
+
if "```json" in response_text:
|
| 230 |
+
json_str = response_text.split("```json")[1].split("```")[0]
|
| 231 |
+
elif "```" in response_text:
|
| 232 |
+
json_str = response_text.split("```")[1].split("```")[0]
|
| 233 |
+
else:
|
| 234 |
+
json_str = response_text
|
| 235 |
+
|
| 236 |
+
action_data = json.loads(json_str)
|
| 237 |
+
except (json.JSONDecodeError, IndexError):
|
| 238 |
+
print(f"Failed to parse LLM response: {response_text[:100]}")
|
| 239 |
+
# Fallback to simple diagnosis
|
| 240 |
+
action_data = {"action_type": "bash_cmd", "command": "ps aux"}
|
| 241 |
+
|
| 242 |
+
except Exception as e:
|
| 243 |
+
print(f"LLM error: {e}. Falling back to heuristic...")
|
| 244 |
+
# Fallback heuristic actions
|
| 245 |
+
if step_count == 1:
|
| 246 |
+
action_data = {"action_type": "bash_cmd", "command": "systemctl status nginx"}
|
| 247 |
+
else:
|
| 248 |
+
action_data = {"action_type": "submit", "summary": "Diagnostics complete"}
|
| 249 |
+
|
| 250 |
+
# Step in environment
|
| 251 |
+
try:
|
| 252 |
+
result = send_request("POST", "/step", json={
|
| 253 |
+
"episode_id": episode_id,
|
| 254 |
+
"action": action_data
|
| 255 |
+
})
|
| 256 |
+
|
| 257 |
+
obs = result["observation"]
|
| 258 |
+
reward = result["reward"]
|
| 259 |
+
done = result["done"]
|
| 260 |
+
|
| 261 |
+
step_count = obs["step_number"]
|
| 262 |
+
total_reward = reward["total_reward"]
|
| 263 |
+
|
| 264 |
+
actions_taken.append(action_data)
|
| 265 |
+
|
| 266 |
+
print(f"\nStep {step_count}/{max_steps}")
|
| 267 |
+
print(f"Action: {action_data['action_type']}")
|
| 268 |
+
if action_data.get("command"):
|
| 269 |
+
print(f"Command: {action_data['command']}")
|
| 270 |
+
elif action_data.get("file_path"):
|
| 271 |
+
print(f"File: {action_data['file_path']}")
|
| 272 |
+
|
| 273 |
+
print(f"Reward: {reward['step_reward']:+.3f} (total: {total_reward:.3f})")
|
| 274 |
+
print(f"Info: {reward['explanation'][:100]}")
|
| 275 |
+
|
| 276 |
+
if done:
|
| 277 |
+
print(f"\n{'='*60}")
|
| 278 |
+
print("EPISODE COMPLETE!")
|
| 279 |
+
print(f"Final Reward: {total_reward:.3f}")
|
| 280 |
+
print(f"Steps taken: {step_count}")
|
| 281 |
+
print(f"{'='*60}\n")
|
| 282 |
+
break
|
| 283 |
+
|
| 284 |
+
except Exception as e:
|
| 285 |
+
print(f"Step error: {e}")
|
| 286 |
+
break
|
| 287 |
+
|
| 288 |
+
# Small delay to avoid rate limiting
|
| 289 |
+
time.sleep(0.5)
|
| 290 |
+
|
| 291 |
+
# Grade the episode
|
| 292 |
+
try:
|
| 293 |
+
grade_result = send_request("POST", "/grader", json={"episode_id": episode_id})
|
| 294 |
+
print(f"\nGrader Results:")
|
| 295 |
+
print(f"Score: {grade_result['score']:.3f}/1.0")
|
| 296 |
+
print(f"Breakdown: {json.dumps(grade_result['breakdown'], indent=2)}")
|
| 297 |
+
print(f"Feedback: {grade_result['feedback']}")
|
| 298 |
+
except Exception as e:
|
| 299 |
+
print(f"Grading error: {e}")
|
| 300 |
+
|
| 301 |
+
return {
|
| 302 |
+
"task_id": task_id,
|
| 303 |
+
"episode_id": episode_id,
|
| 304 |
+
"final_reward": total_reward,
|
| 305 |
+
"step_count": step_count,
|
| 306 |
+
"actions": actions_taken,
|
| 307 |
+
}
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
def main():
|
| 311 |
+
parser = argparse.ArgumentParser(description="Run DevOpsEnv baseline agent")
|
| 312 |
+
parser.add_argument("--task", default="task1", help="Task ID (task1, task2, or task3)")
|
| 313 |
+
parser.add_argument(
|
| 314 |
+
"--model",
|
| 315 |
+
default=None,
|
| 316 |
+
help=(
|
| 317 |
+
"Model name (default: env var MODEL_NAME). "
|
| 318 |
+
"For Gemini, pass a real model ID like gemini-3-flash-preview "
|
| 319 |
+
"or use --model gemini to auto-resolve to GEMINI_MODEL."
|
| 320 |
+
),
|
| 321 |
+
)
|
| 322 |
+
parser.add_argument("--api-url", default=None, help="API URL (default: env var API_BASE_URL)")
|
| 323 |
+
parser.add_argument("--hf-token", default=None, help="HF token (default: env var HF_TOKEN)")
|
| 324 |
+
parser.add_argument(
|
| 325 |
+
"--openai-base-url",
|
| 326 |
+
default=None,
|
| 327 |
+
help="OpenAI-compatible base URL for non-OpenAI providers (for example Gemini OpenAI API)",
|
| 328 |
+
)
|
| 329 |
+
|
| 330 |
+
args = parser.parse_args()
|
| 331 |
+
|
| 332 |
+
# Override env variables if provided
|
| 333 |
+
global API_BASE_URL, MODEL_NAME, HF_TOKEN, OPENAI_BASE_URL
|
| 334 |
+
if args.api_url:
|
| 335 |
+
API_BASE_URL = args.api_url
|
| 336 |
+
if args.model:
|
| 337 |
+
MODEL_NAME = args.model
|
| 338 |
+
if args.hf_token:
|
| 339 |
+
HF_TOKEN = args.hf_token
|
| 340 |
+
if args.openai_base_url:
|
| 341 |
+
OPENAI_BASE_URL = args.openai_base_url
|
| 342 |
+
|
| 343 |
+
try:
|
| 344 |
+
result = run_agent(args.task, model=MODEL_NAME)
|
| 345 |
+
print(json.dumps(result, indent=2))
|
| 346 |
+
except Exception as e:
|
| 347 |
+
print(f"Fatal error: {e}", file=sys.stderr)
|
| 348 |
+
sys.exit(1)
|
| 349 |
+
|
| 350 |
+
|
| 351 |
+
if __name__ == "__main__":
|
| 352 |
+
main()
|
models.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
"""
|
| 2 |
-
Pydantic models for
|
| 3 |
|
| 4 |
-
Domain:
|
| 5 |
"""
|
| 6 |
from __future__ import annotations
|
| 7 |
|
|
@@ -10,20 +10,21 @@ from pydantic import BaseModel, Field
|
|
| 10 |
|
| 11 |
|
| 12 |
# ---------------------------------------------------------------------------
|
| 13 |
-
#
|
| 14 |
# ---------------------------------------------------------------------------
|
| 15 |
|
| 16 |
-
class
|
| 17 |
-
"""
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
|
|
|
| 27 |
|
| 28 |
|
| 29 |
# ---------------------------------------------------------------------------
|
|
@@ -32,23 +33,18 @@ class TicketInfo(BaseModel):
|
|
| 32 |
|
| 33 |
class Observation(BaseModel):
|
| 34 |
"""Everything the agent sees at each step."""
|
| 35 |
-
task_id: str = Field(description="
|
| 36 |
task_description: str = Field(description="Human-readable task description")
|
| 37 |
episode_id: str = Field(description="Unique episode UUID")
|
| 38 |
-
|
| 39 |
thread_history: List[Dict[str, str]] = Field(
|
| 40 |
default_factory=list,
|
| 41 |
description="Ordered list of {'role': 'agent'|'system', 'content': str}"
|
| 42 |
)
|
| 43 |
-
available_actions: List[str]
|
| 44 |
-
description="List of valid action_type values for this step"
|
| 45 |
-
)
|
| 46 |
step_number: int
|
| 47 |
max_steps: int
|
| 48 |
-
hint: Optional[str] = Field(
|
| 49 |
-
default=None,
|
| 50 |
-
description="Optional hint shown to the agent (may be None)"
|
| 51 |
-
)
|
| 52 |
|
| 53 |
|
| 54 |
# ---------------------------------------------------------------------------
|
|
@@ -56,68 +52,12 @@ class Observation(BaseModel):
|
|
| 56 |
# ---------------------------------------------------------------------------
|
| 57 |
|
| 58 |
class Action(BaseModel):
|
| 59 |
-
"""
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
""
|
| 64 |
-
|
| 65 |
-
description=(
|
| 66 |
-
"One of: classify | extract | respond | escalate | resolve | submit"
|
| 67 |
-
)
|
| 68 |
-
)
|
| 69 |
-
|
| 70 |
-
# ---- Task 1: classify -------------------------------------------------
|
| 71 |
-
category: Optional[str] = Field(
|
| 72 |
-
default=None,
|
| 73 |
-
description=(
|
| 74 |
-
"Ticket category: billing | technical | account | "
|
| 75 |
-
"feature_request | complaint | general"
|
| 76 |
-
)
|
| 77 |
-
)
|
| 78 |
-
priority: Optional[str] = Field(
|
| 79 |
-
default=None,
|
| 80 |
-
description="Priority level: low | medium | high | critical"
|
| 81 |
-
)
|
| 82 |
-
|
| 83 |
-
# ---- Task 2: extract --------------------------------------------------
|
| 84 |
-
extracted_entities: Optional[Dict[str, Any]] = Field(
|
| 85 |
-
default=None,
|
| 86 |
-
description=(
|
| 87 |
-
"Key-value pairs extracted from the ticket. Expected keys vary by "
|
| 88 |
-
"ticket; see task description."
|
| 89 |
-
)
|
| 90 |
-
)
|
| 91 |
-
required_actions: Optional[List[str]] = Field(
|
| 92 |
-
default=None,
|
| 93 |
-
description="List of actions that must be taken to resolve the ticket"
|
| 94 |
-
)
|
| 95 |
-
|
| 96 |
-
# ---- Task 3: respond / resolve ----------------------------------------
|
| 97 |
-
response_text: Optional[str] = Field(
|
| 98 |
-
default=None,
|
| 99 |
-
description="Full text of the agent's response to the customer"
|
| 100 |
-
)
|
| 101 |
-
resolution_steps: Optional[List[str]] = Field(
|
| 102 |
-
default=None,
|
| 103 |
-
description="Ordered list of steps to resolve the ticket"
|
| 104 |
-
)
|
| 105 |
-
|
| 106 |
-
# ---- escalate ---------------------------------------------------------
|
| 107 |
-
escalation_team: Optional[str] = Field(
|
| 108 |
-
default=None,
|
| 109 |
-
description=(
|
| 110 |
-
"Team to escalate to: billing_team | engineering | "
|
| 111 |
-
"account_management | legal"
|
| 112 |
-
)
|
| 113 |
-
)
|
| 114 |
-
escalation_reason: Optional[str] = Field(
|
| 115 |
-
default=None,
|
| 116 |
-
description="Brief reason for escalation"
|
| 117 |
-
)
|
| 118 |
-
|
| 119 |
-
# ---- submit -----------------------------------------------------------
|
| 120 |
-
# No extra fields — signals the agent is done with the episode.
|
| 121 |
|
| 122 |
|
| 123 |
# ---------------------------------------------------------------------------
|
|
@@ -126,19 +66,13 @@ class Action(BaseModel):
|
|
| 126 |
|
| 127 |
class Reward(BaseModel):
|
| 128 |
"""Per-step reward signal."""
|
| 129 |
-
step_reward: float
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
total_reward: float = Field(
|
| 133 |
-
description="Cumulative reward for the episode so far"
|
| 134 |
-
)
|
| 135 |
-
explanation: str = Field(
|
| 136 |
-
description="Human-readable explanation of what drove the reward"
|
| 137 |
-
)
|
| 138 |
|
| 139 |
|
| 140 |
# ---------------------------------------------------------------------------
|
| 141 |
-
# Step
|
| 142 |
# ---------------------------------------------------------------------------
|
| 143 |
|
| 144 |
class StepResult(BaseModel):
|
|
@@ -149,7 +83,7 @@ class StepResult(BaseModel):
|
|
| 149 |
|
| 150 |
|
| 151 |
# ---------------------------------------------------------------------------
|
| 152 |
-
# State
|
| 153 |
# ---------------------------------------------------------------------------
|
| 154 |
|
| 155 |
class State(BaseModel):
|
|
@@ -159,53 +93,39 @@ class State(BaseModel):
|
|
| 159 |
max_steps: int
|
| 160 |
done: bool
|
| 161 |
total_reward: float
|
| 162 |
-
history: List[Dict[str, Any]] = Field(
|
| 163 |
-
|
| 164 |
-
description="Ordered list of (action, reward) pairs for the episode"
|
| 165 |
-
)
|
| 166 |
-
final_score: Optional[float] = Field(
|
| 167 |
-
default=None,
|
| 168 |
-
description="Grader score 0.0–1.0, set when episode is done"
|
| 169 |
-
)
|
| 170 |
|
| 171 |
|
| 172 |
# ---------------------------------------------------------------------------
|
| 173 |
-
# Task
|
| 174 |
# ---------------------------------------------------------------------------
|
| 175 |
|
| 176 |
class TaskInfo(BaseModel):
|
| 177 |
task_id: str
|
| 178 |
name: str
|
| 179 |
description: str
|
| 180 |
-
difficulty: str
|
| 181 |
max_steps: int
|
| 182 |
-
action_schema: Dict[str, Any] = Field(
|
| 183 |
-
description="JSON Schema fragment describing required Action fields"
|
| 184 |
-
)
|
| 185 |
|
| 186 |
|
| 187 |
# ---------------------------------------------------------------------------
|
| 188 |
-
#
|
| 189 |
# ---------------------------------------------------------------------------
|
| 190 |
|
| 191 |
class GraderResponse(BaseModel):
|
| 192 |
episode_id: str
|
| 193 |
task_id: str
|
| 194 |
score: float = Field(description="Final grader score 0.0–1.0")
|
| 195 |
-
breakdown: Dict[str, float] = Field(
|
| 196 |
-
default_factory=dict,
|
| 197 |
-
description="Per-criterion partial scores"
|
| 198 |
-
)
|
| 199 |
feedback: str
|
| 200 |
|
| 201 |
|
| 202 |
-
class BaselineScore(BaseModel):
|
| 203 |
-
task_id: str
|
| 204 |
-
score: float
|
| 205 |
-
details: Dict[str, Any] = Field(default_factory=dict)
|
| 206 |
-
|
| 207 |
-
|
| 208 |
class BaselineResult(BaseModel):
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
"""
|
| 2 |
+
Pydantic models for DevOpsEnv OpenEnv environment.
|
| 3 |
|
| 4 |
+
Domain: Linux DevOps & SRE Troubleshooting
|
| 5 |
"""
|
| 6 |
from __future__ import annotations
|
| 7 |
|
|
|
|
| 10 |
|
| 11 |
|
| 12 |
# ---------------------------------------------------------------------------
|
| 13 |
+
# System State Models
|
| 14 |
# ---------------------------------------------------------------------------
|
| 15 |
|
| 16 |
+
class SystemState(BaseModel):
|
| 17 |
+
"""Current state of the mock Linux server."""
|
| 18 |
+
task_id: str
|
| 19 |
+
available_commands: List[str]
|
| 20 |
+
filesystem_snapshot: str
|
| 21 |
+
running_processes: List[Dict[str, Any]]
|
| 22 |
+
service_status: Dict[str, str]
|
| 23 |
+
logs: str
|
| 24 |
+
http_ports_open: List[int]
|
| 25 |
+
docker_containers: List[Dict[str, str]]
|
| 26 |
+
cpu_usage: float
|
| 27 |
+
memory_usage_mb: int
|
| 28 |
|
| 29 |
|
| 30 |
# ---------------------------------------------------------------------------
|
|
|
|
| 33 |
|
| 34 |
class Observation(BaseModel):
|
| 35 |
"""Everything the agent sees at each step."""
|
| 36 |
+
task_id: str = Field(description="task1 | task2 | task3")
|
| 37 |
task_description: str = Field(description="Human-readable task description")
|
| 38 |
episode_id: str = Field(description="Unique episode UUID")
|
| 39 |
+
system_state: SystemState
|
| 40 |
thread_history: List[Dict[str, str]] = Field(
|
| 41 |
default_factory=list,
|
| 42 |
description="Ordered list of {'role': 'agent'|'system', 'content': str}"
|
| 43 |
)
|
| 44 |
+
available_actions: List[str]
|
|
|
|
|
|
|
| 45 |
step_number: int
|
| 46 |
max_steps: int
|
| 47 |
+
hint: Optional[str] = Field(default=None)
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
|
| 50 |
# ---------------------------------------------------------------------------
|
|
|
|
| 52 |
# ---------------------------------------------------------------------------
|
| 53 |
|
| 54 |
class Action(BaseModel):
|
| 55 |
+
"""Agent action: run a bash command, edit a file, or submit."""
|
| 56 |
+
action_type: str = Field(description="bash_cmd | file_edit | submit")
|
| 57 |
+
command: Optional[str] = Field(default=None, description="Bash command to execute")
|
| 58 |
+
file_path: Optional[str] = Field(default=None, description="Absolute path to file to edit")
|
| 59 |
+
file_content: Optional[str] = Field(default=None, description="New full content for the file")
|
| 60 |
+
summary: Optional[str] = Field(default=None, description="Final summary of actions taken")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
|
| 63 |
# ---------------------------------------------------------------------------
|
|
|
|
| 66 |
|
| 67 |
class Reward(BaseModel):
|
| 68 |
"""Per-step reward signal."""
|
| 69 |
+
step_reward: float
|
| 70 |
+
total_reward: float
|
| 71 |
+
explanation: str
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
|
| 74 |
# ---------------------------------------------------------------------------
|
| 75 |
+
# Step Result
|
| 76 |
# ---------------------------------------------------------------------------
|
| 77 |
|
| 78 |
class StepResult(BaseModel):
|
|
|
|
| 83 |
|
| 84 |
|
| 85 |
# ---------------------------------------------------------------------------
|
| 86 |
+
# State
|
| 87 |
# ---------------------------------------------------------------------------
|
| 88 |
|
| 89 |
class State(BaseModel):
|
|
|
|
| 93 |
max_steps: int
|
| 94 |
done: bool
|
| 95 |
total_reward: float
|
| 96 |
+
history: List[Dict[str, Any]] = Field(default_factory=list)
|
| 97 |
+
final_score: Optional[float] = Field(default=None)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
|
| 100 |
# ---------------------------------------------------------------------------
|
| 101 |
+
# Task Metadata
|
| 102 |
# ---------------------------------------------------------------------------
|
| 103 |
|
| 104 |
class TaskInfo(BaseModel):
|
| 105 |
task_id: str
|
| 106 |
name: str
|
| 107 |
description: str
|
| 108 |
+
difficulty: str
|
| 109 |
max_steps: int
|
|
|
|
|
|
|
|
|
|
| 110 |
|
| 111 |
|
| 112 |
# ---------------------------------------------------------------------------
|
| 113 |
+
# Grader Response
|
| 114 |
# ---------------------------------------------------------------------------
|
| 115 |
|
| 116 |
class GraderResponse(BaseModel):
|
| 117 |
episode_id: str
|
| 118 |
task_id: str
|
| 119 |
score: float = Field(description="Final grader score 0.0–1.0")
|
| 120 |
+
breakdown: Dict[str, float] = Field(default_factory=dict)
|
|
|
|
|
|
|
|
|
|
| 121 |
feedback: str
|
| 122 |
|
| 123 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
class BaselineResult(BaseModel):
|
| 125 |
+
"""Result of running the baseline agent."""
|
| 126 |
+
task_id: str
|
| 127 |
+
episode_id: str
|
| 128 |
+
final_score: float
|
| 129 |
+
step_count: int
|
| 130 |
+
total_reward: float
|
| 131 |
+
actions: List[Dict[str, Any]]
|
requirements.txt
CHANGED
|
@@ -3,4 +3,6 @@ uvicorn[standard]>=0.30.0
|
|
| 3 |
pydantic>=2.7.0
|
| 4 |
openai>=1.35.0
|
| 5 |
httpx>=0.27.0
|
| 6 |
-
python-multipart>=0.0.9
|
|
|
|
|
|
|
|
|
| 3 |
pydantic>=2.7.0
|
| 4 |
openai>=1.35.0
|
| 5 |
httpx>=0.27.0
|
| 6 |
+
python-multipart>=0.0.9
|
| 7 |
+
requests>=2.31.0
|
| 8 |
+
google-genai>=1.15.0
|
test_integration.py
ADDED
|
@@ -0,0 +1,116 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Quick integration test for DevOpsEnv.
|
| 3 |
+
|
| 4 |
+
This runs a full episode for each task to verify everything works.
|
| 5 |
+
"""
|
| 6 |
+
import environment as env
|
| 7 |
+
from models import Action
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def test_task(task_id: str, max_test_steps: int = 5):
|
| 11 |
+
"""Test a single task."""
|
| 12 |
+
print(f"\n{'='*60}")
|
| 13 |
+
print(f"Testing {task_id}")
|
| 14 |
+
print(f"{'='*60}")
|
| 15 |
+
|
| 16 |
+
# Reset
|
| 17 |
+
print("1. Calling reset()...")
|
| 18 |
+
obs = env.reset(task_id)
|
| 19 |
+
episode_id = obs.episode_id
|
| 20 |
+
print(f"[OK] Episode created: {episode_id}")
|
| 21 |
+
print(f" Task: {obs.task_description[:80]}...")
|
| 22 |
+
print(f" Max steps: {obs.max_steps}")
|
| 23 |
+
print(f" System state: cpu={obs.system_state.cpu_usage:.1f}%, mem={obs.system_state.memory_usage_mb}MB")
|
| 24 |
+
|
| 25 |
+
# Take steps
|
| 26 |
+
print(f"\n2. Taking {max_test_steps} steps...")
|
| 27 |
+
for i in range(max_test_steps):
|
| 28 |
+
if task_id == "task1":
|
| 29 |
+
if i == 0:
|
| 30 |
+
action = Action(action_type="bash_cmd", command="systemctl status nginx")
|
| 31 |
+
elif i == 1:
|
| 32 |
+
action = Action(action_type="bash_cmd", command="systemctl try-restart nginx")
|
| 33 |
+
elif i == 2:
|
| 34 |
+
action = Action(action_type="bash_cmd", command="nginx -t")
|
| 35 |
+
else:
|
| 36 |
+
action = Action(action_type="bash_cmd", command="curl http://localhost")
|
| 37 |
+
elif task_id == "task2":
|
| 38 |
+
if i == 0:
|
| 39 |
+
action = Action(action_type="bash_cmd", command="cat /srv/docker-compose.yml")
|
| 40 |
+
elif i == 1:
|
| 41 |
+
action = Action(
|
| 42 |
+
action_type="file_edit",
|
| 43 |
+
file_path="/srv/docker-compose.yml",
|
| 44 |
+
file_content="version: '3.8'\nservices:\n mockapi:\n image: mockapi:latest\n ports:\n - \"3000:3000\""
|
| 45 |
+
)
|
| 46 |
+
elif i == 2:
|
| 47 |
+
action = Action(action_type="bash_cmd", command="docker-compose up -d")
|
| 48 |
+
else:
|
| 49 |
+
action = Action(action_type="bash_cmd", command="docker ps")
|
| 50 |
+
else: # task3
|
| 51 |
+
if i == 0:
|
| 52 |
+
action = Action(action_type="bash_cmd", command="ps aux | grep python")
|
| 53 |
+
elif i == 1:
|
| 54 |
+
action = Action(action_type="bash_cmd", command="kill 300")
|
| 55 |
+
elif i == 2:
|
| 56 |
+
action = Action(
|
| 57 |
+
action_type="file_edit",
|
| 58 |
+
file_path="/opt/mockapi/app.py",
|
| 59 |
+
file_content="import json\nfrom flask import Flask\n\napp = Flask(__name__)\n\n@app.route('/api/data')\ndef get_data():\n return json.dumps({'status': 'ok'})\n\nif __name__ == '__main__':\n app.run()\n"
|
| 60 |
+
)
|
| 61 |
+
else:
|
| 62 |
+
action = Action(action_type="bash_cmd", command="python3 /opt/mockapi/app.py &")
|
| 63 |
+
|
| 64 |
+
try:
|
| 65 |
+
result = env.step(episode_id, action)
|
| 66 |
+
print(f" Step {i+1}: {action.action_type} - Reward: {result.reward.step_reward:+.3f}")
|
| 67 |
+
if result.done:
|
| 68 |
+
print(f" -> Episode completed early")
|
| 69 |
+
break
|
| 70 |
+
except Exception as e:
|
| 71 |
+
print(f" Step {i+1} ERROR: {e}")
|
| 72 |
+
break
|
| 73 |
+
|
| 74 |
+
# Check state
|
| 75 |
+
print(f"\n3. Calling get_state()...")
|
| 76 |
+
state = env.get_state(episode_id)
|
| 77 |
+
print(f"[OK] State: step_number={state.step_number}, total_reward={state.total_reward:.3f}, done={state.done}")
|
| 78 |
+
|
| 79 |
+
# Finish episode if not already done
|
| 80 |
+
if not state.done:
|
| 81 |
+
print(f"\n4. Calling submit()...")
|
| 82 |
+
result = env.step(episode_id, Action(action_type="submit"))
|
| 83 |
+
print(f"[OK] Episode submitted, done={result.done}")
|
| 84 |
+
|
| 85 |
+
# Grade
|
| 86 |
+
print(f"\n5. Calling grade()...")
|
| 87 |
+
try:
|
| 88 |
+
score, breakdown, feedback = env.grade(episode_id)
|
| 89 |
+
print(f"[OK] Score: {score:.3f}/1.0")
|
| 90 |
+
print(f" Breakdown: {breakdown}")
|
| 91 |
+
print(f" Feedback: {feedback}")
|
| 92 |
+
except Exception as e:
|
| 93 |
+
print(f"[ERROR] Grading error: {e}")
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def main():
|
| 97 |
+
"""Run all tests."""
|
| 98 |
+
print("DevOpsEnv Integration Test")
|
| 99 |
+
print("="*60)
|
| 100 |
+
|
| 101 |
+
try:
|
| 102 |
+
test_task("task1", max_test_steps=5)
|
| 103 |
+
test_task("task2", max_test_steps=5)
|
| 104 |
+
test_task("task3", max_test_steps=5)
|
| 105 |
+
|
| 106 |
+
print(f"\n{'='*60}")
|
| 107 |
+
print("[OK] All tests completed successfully!")
|
| 108 |
+
print(f"{'='*60}")
|
| 109 |
+
except Exception as e:
|
| 110 |
+
print(f"\n[ERROR] Test failed: {e}")
|
| 111 |
+
import traceback
|
| 112 |
+
traceback.print_exc()
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
if __name__ == "__main__":
|
| 116 |
+
main()
|
tests_new.py
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Comprehensive tests for DevOpsEnv.
|
| 3 |
+
|
| 4 |
+
Run with: pytest tests/test_environment.py -v
|
| 5 |
+
"""
|
| 6 |
+
import pytest
|
| 7 |
+
import json
|
| 8 |
+
from unittest.mock import patch
|
| 9 |
+
|
| 10 |
+
import environment as env
|
| 11 |
+
from models import Action, Observation, StepResult, State
|
| 12 |
+
from data import TASK_META
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class TestReset:
|
| 16 |
+
"""Test episode reset functionality."""
|
| 17 |
+
|
| 18 |
+
def test_reset_valid_task(self):
|
| 19 |
+
"""Reset creates a valid episode."""
|
| 20 |
+
obs = env.reset("task1")
|
| 21 |
+
|
| 22 |
+
assert isinstance(obs, Observation)
|
| 23 |
+
assert obs.task_id == "task1"
|
| 24 |
+
assert obs.episode_id is not None
|
| 25 |
+
assert len(obs.episode_id) > 0
|
| 26 |
+
assert obs.system_state is not None
|
| 27 |
+
assert obs.step_number == 0
|
| 28 |
+
assert obs.max_steps == TASK_META["task1"]["max_steps"]
|
| 29 |
+
|
| 30 |
+
def test_reset_invalid_task(self):
|
| 31 |
+
"""Reset raises error for unknown task."""
|
| 32 |
+
with pytest.raises(ValueError):
|
| 33 |
+
env.reset("invalid_task")
|
| 34 |
+
|
| 35 |
+
def test_reset_creates_episode_state(self):
|
| 36 |
+
"""Reset creates episode in internal state."""
|
| 37 |
+
obs = env.reset("task2")
|
| 38 |
+
|
| 39 |
+
assert obs.episode_id in env._EPISODES
|
| 40 |
+
ep = env._EPISODES[obs.episode_id]
|
| 41 |
+
assert ep["task_id"] == "task2"
|
| 42 |
+
assert ep["step_number"] == 0
|
| 43 |
+
assert ep["done"] is False
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
class TestStep:
|
| 47 |
+
"""Test step execution."""
|
| 48 |
+
|
| 49 |
+
def test_step_bash_command(self):
|
| 50 |
+
"""Step handles bash_cmd action."""
|
| 51 |
+
obs = env.reset("task1")
|
| 52 |
+
|
| 53 |
+
action = Action(action_type="bash_cmd", command="systemctl status nginx")
|
| 54 |
+
result = env.step(obs.episode_id, action)
|
| 55 |
+
|
| 56 |
+
assert isinstance(result, StepResult)
|
| 57 |
+
assert result.observation.step_number == 1
|
| 58 |
+
assert result.reward.step_reward != 0
|
| 59 |
+
assert result.done is False
|
| 60 |
+
|
| 61 |
+
def test_step_file_edit(self):
|
| 62 |
+
"""Step handles file_edit action."""
|
| 63 |
+
obs = env.reset("task2")
|
| 64 |
+
|
| 65 |
+
action = Action(
|
| 66 |
+
action_type="file_edit",
|
| 67 |
+
file_path="/srv/docker-compose.yml",
|
| 68 |
+
file_content="version: '3.8'\nservices:\n test: {}"
|
| 69 |
+
)
|
| 70 |
+
result = env.step(obs.episode_id, action)
|
| 71 |
+
|
| 72 |
+
assert isinstance(result, StepResult)
|
| 73 |
+
assert result.observation.step_number == 1
|
| 74 |
+
|
| 75 |
+
def test_step_submit(self):
|
| 76 |
+
"""Step with submit action marks episode done."""
|
| 77 |
+
obs = env.reset("task1")
|
| 78 |
+
|
| 79 |
+
action = Action(action_type="submit", summary="Done")
|
| 80 |
+
result = env.step(obs.episode_id, action)
|
| 81 |
+
|
| 82 |
+
assert result.done is True
|
| 83 |
+
assert result.observation.available_actions == []
|
| 84 |
+
|
| 85 |
+
def test_step_invalid_episode(self):
|
| 86 |
+
"""Step raises error for invalid episode."""
|
| 87 |
+
action = Action(action_type="bash_cmd", command="ls")
|
| 88 |
+
|
| 89 |
+
with pytest.raises(KeyError):
|
| 90 |
+
env.step("invalid_episode_id", action)
|
| 91 |
+
|
| 92 |
+
def test_step_after_done(self):
|
| 93 |
+
"""Step raises error after episode is done."""
|
| 94 |
+
obs = env.reset("task1")
|
| 95 |
+
|
| 96 |
+
# End the episode
|
| 97 |
+
action1 = Action(action_type="submit")
|
| 98 |
+
env.step(obs.episode_id, action1)
|
| 99 |
+
|
| 100 |
+
# Try to step again
|
| 101 |
+
with pytest.raises(ValueError):
|
| 102 |
+
env.step(obs.episode_id, action1)
|
| 103 |
+
|
| 104 |
+
def test_step_max_steps_limit(self):
|
| 105 |
+
"""Episode ends after max_steps."""
|
| 106 |
+
obs = env.reset("task1")
|
| 107 |
+
max_steps = obs.max_steps
|
| 108 |
+
|
| 109 |
+
for i in range(max_steps):
|
| 110 |
+
action = Action(action_type="bash_cmd", command="ps aux")
|
| 111 |
+
result = env.step(obs.episode_id, action)
|
| 112 |
+
|
| 113 |
+
if i < max_steps - 1:
|
| 114 |
+
assert result.done is False
|
| 115 |
+
else:
|
| 116 |
+
assert result.done is True
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
class TestState:
|
| 120 |
+
"""Test state retrieval."""
|
| 121 |
+
|
| 122 |
+
def test_get_state(self):
|
| 123 |
+
"""get_state returns current episode state."""
|
| 124 |
+
obs = env.reset("task3")
|
| 125 |
+
|
| 126 |
+
state = env.get_state(obs.episode_id)
|
| 127 |
+
|
| 128 |
+
assert isinstance(state, State)
|
| 129 |
+
assert state.episode_id == obs.episode_id
|
| 130 |
+
assert state.task_id == "task3"
|
| 131 |
+
assert state.step_number == 0
|
| 132 |
+
assert state.done is False
|
| 133 |
+
|
| 134 |
+
def test_get_state_invalid_episode(self):
|
| 135 |
+
"""get_state raises error for invalid episode."""
|
| 136 |
+
with pytest.raises(KeyError):
|
| 137 |
+
env.get_state("invalid_id")
|
| 138 |
+
|
| 139 |
+
def test_state_history(self):
|
| 140 |
+
"""State includes action history."""
|
| 141 |
+
obs = env.reset("task1")
|
| 142 |
+
|
| 143 |
+
# Take Actions
|
| 144 |
+
action1 = Action(action_type="bash_cmd", command="ps aux")
|
| 145 |
+
env.step(obs.episode_id, action1)
|
| 146 |
+
|
| 147 |
+
state = env.get_state(obs.episode_id)
|
| 148 |
+
|
| 149 |
+
assert len(state.history) == 1
|
| 150 |
+
assert state.history[0]["action_type"] == "bash_cmd"
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
class TestGrading:
|
| 154 |
+
"""Test episode grading."""
|
| 155 |
+
|
| 156 |
+
def test_grade_task1_nginx_running(self):
|
| 157 |
+
"""Task 1 grades based on nginx status."""
|
| 158 |
+
obs = env.reset("task1")
|
| 159 |
+
|
| 160 |
+
# Run commands to fix nginx
|
| 161 |
+
env.step(obs.episode_id, Action(action_type="bash_cmd", command="systemctl restart nginx"))
|
| 162 |
+
env.step(obs.episode_id, Action(action_type="bash_cmd", command="nginx -t"))
|
| 163 |
+
env.step(obs.episode_id, Action(action_type="bash_cmd", command="curl http://localhost"))
|
| 164 |
+
env.step(obs.episode_id, Action(action_type="submit"))
|
| 165 |
+
|
| 166 |
+
score, breakdown, feedback = env.grade(obs.episode_id)
|
| 167 |
+
|
| 168 |
+
assert 0.0 <= score <= 1.0
|
| 169 |
+
assert "nginx_running" in breakdown
|
| 170 |
+
assert "config_valid" in breakdown
|
| 171 |
+
assert "http_200" in breakdown
|
| 172 |
+
|
| 173 |
+
def test_grade_invalid_episode(self):
|
| 174 |
+
"""grade raises error for invalid episode."""
|
| 175 |
+
with pytest.raises(KeyError):
|
| 176 |
+
env.grade("invalid_id")
|
| 177 |
+
|
| 178 |
+
def test_grade_not_done(self):
|
| 179 |
+
"""grade raises error if episode not done."""
|
| 180 |
+
obs = env.reset("task1")
|
| 181 |
+
# Don't finish the episode
|
| 182 |
+
|
| 183 |
+
with pytest.raises(ValueError):
|
| 184 |
+
env.grade(obs.episode_id)
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
class TestSystemSimulation:
|
| 188 |
+
"""Test mock system state simulation."""
|
| 189 |
+
|
| 190 |
+
def test_task1_initial_state(self):
|
| 191 |
+
"""Task 1 initializes with nginx crashed."""
|
| 192 |
+
obs = env.reset("task1")
|
| 193 |
+
|
| 194 |
+
assert obs.system_state.service_status.get("nginx") == "inactive"
|
| 195 |
+
assert 80 not in obs.system_state.http_ports_open
|
| 196 |
+
|
| 197 |
+
def test_task2_initial_state(self):
|
| 198 |
+
"""Task 2 initializes with docker misconfigured."""
|
| 199 |
+
obs = env.reset("task2")
|
| 200 |
+
|
| 201 |
+
assert obs.system_state.service_status.get("docker") == "active"
|
| 202 |
+
assert 80 in obs.system_state.http_ports_open
|
| 203 |
+
|
| 204 |
+
def test_task3_initial_state(self):
|
| 205 |
+
"""Task 3 initializes with memory leak."""
|
| 206 |
+
obs = env.reset("task3")
|
| 207 |
+
|
| 208 |
+
assert obs.system_state.service_status.get("mockapi") == "active"
|
| 209 |
+
# Should have high memory usage
|
| 210 |
+
assert obs.system_state.memory_usage_mb > 1024
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
class TestRewards:
|
| 214 |
+
"""Test reward calculation."""
|
| 215 |
+
|
| 216 |
+
def test_step_reward_positive(self):
|
| 217 |
+
"""Taking actions yields positive reward."""
|
| 218 |
+
obs = env.reset("task1")
|
| 219 |
+
|
| 220 |
+
action = Action(action_type="bash_cmd", command="ps aux")
|
| 221 |
+
result = env.step(obs.episode_id, action)
|
| 222 |
+
|
| 223 |
+
assert result.reward.step_reward > -1.0 # Not all negative
|
| 224 |
+
|
| 225 |
+
def test_total_reward_accumulation(self):
|
| 226 |
+
"""Total reward accumulates across steps."""
|
| 227 |
+
obs = env.reset("task1")
|
| 228 |
+
|
| 229 |
+
env.step(obs.episode_id, Action(action_type="bash_cmd", command="ps aux"))
|
| 230 |
+
result1 = env.step(obs.episode_id, Action(action_type="bash_cmd", command="ls"))
|
| 231 |
+
total1 = result1.reward.total_reward
|
| 232 |
+
|
| 233 |
+
result2 = env.step(obs.episode_id, Action(action_type="bash_cmd", command="pwd"))
|
| 234 |
+
total2 = result2.reward.total_reward
|
| 235 |
+
|
| 236 |
+
# Total reward should accumulate
|
| 237 |
+
assert total2 >= total1 or total2 < total1 # Can go either way depending on grader
|
| 238 |
+
|
| 239 |
+
|
| 240 |
+
@pytest.fixture(autouse=True)
|
| 241 |
+
def cleanup():
|
| 242 |
+
"""Clean up episodes after each test."""
|
| 243 |
+
yield
|
| 244 |
+
env._EPISODES.clear()
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
if __name__ == "__main__":
|
| 248 |
+
pytest.main([__file__, "-v"])
|