Spaces:
Sleeping
Sleeping
Commit ·
5e2a88b
1
Parent(s): cd45cc6
Add tasks 4-5, fix metadata, fix port, expand tests to 94
Browse files- README.md +13 -7
- db_er/models.py +1 -1
- inference.py +8 -2
- openenv.yaml +1 -1
- server/environment.py +25 -7
- server/fixtures.py +408 -2
- server/grader.py +33 -0
- tests/test_dumb_agent.py +71 -0
README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
---
|
| 2 |
title: DB-ER Database Emergency Response
|
| 3 |
-
emoji:
|
| 4 |
colorFrom: red
|
| 5 |
colorTo: indigo
|
| 6 |
sdk: docker
|
|
@@ -30,7 +30,7 @@ DB-ER puts an agent in the role of an on-call database engineer responding to a
|
|
| 30 |
|
| 31 |
The catch: the incident logs are sometimes misleading, the database schema has no documentation, and every destructive action is permanent within the episode.
|
| 32 |
|
| 33 |
-
## The
|
| 34 |
|
| 35 |
**Task 1 - Phantom Duplicates (Easy)**
|
| 36 |
A bulk import script ran twice. The `users` table has duplicate email rows that are causing constraint failures across the app. Find them and remove the duplicates while keeping the original records.
|
|
@@ -38,9 +38,15 @@ A bulk import script ran twice. The `users` table has duplicate email rows that
|
|
| 38 |
**Task 2 - Cascading Failure (Medium)**
|
| 39 |
Someone ran a cleanup script that deleted three products still referenced by active purchases. Every checkout is now broken. The `admin_audit_logs` table has the deleted records stored as JSON. Restore them.
|
| 40 |
|
| 41 |
-
**Task 3 - Payroll Black Hole (Hard)**
|
| 42 |
A migration script died partway through. The incident log blames a timeout, but that is a red herring. The real problem is corrupted duplicate employee IDs in the source table that caused the crash. Clean the corruption, then finish the migration and preserve every salary mapping correctly.
|
| 43 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
---
|
| 45 |
|
| 46 |
## How Scoring Works
|
|
@@ -64,7 +70,7 @@ Each episode ends when the agent calls `submit_resolution` or runs out of budget
|
|
| 64 |
|
| 65 |
```bash
|
| 66 |
pip install -r requirements.txt
|
| 67 |
-
uvicorn server.app:app --host 0.0.0.0 --port
|
| 68 |
```
|
| 69 |
|
| 70 |
Run the tests:
|
|
@@ -91,7 +97,7 @@ Follows the standard OpenEnv spec:
|
|
| 91 |
| Endpoint | Method | Description |
|
| 92 |
|----------|--------|-------------|
|
| 93 |
| `/health` | GET | Liveness check |
|
| 94 |
-
| `/reset` | POST | Start a new episode. Pass `task_id` (1
|
| 95 |
| `/step` | POST | Execute one action |
|
| 96 |
| `/state` | GET | Current episode state |
|
| 97 |
| `/schema` | GET | Action and observation schemas |
|
|
@@ -121,7 +127,7 @@ curl -X POST https://aadi-joshi-openenvhack.hf.space/step \
|
|
| 121 |
```
|
| 122 |
db_er/ Pydantic models (action, observation, state) and HTTP client
|
| 123 |
server/ Environment logic, fixtures, grader, and safety guards
|
| 124 |
-
tests/
|
| 125 |
inference.py Baseline agent using the OpenAI client
|
| 126 |
openenv.yaml OpenEnv manifest
|
| 127 |
Dockerfile Container config for HF Spaces (port 7860)
|
|
@@ -137,7 +143,7 @@ Dockerfile Container config for HF Spaces (port 7860)
|
|
| 137 |
| `MODEL_NAME` | Model identifier (required for inference) |
|
| 138 |
| `HF_TOKEN` | API key (required for inference) |
|
| 139 |
| `ENV_BASE_URL` | Point at a remote server instead of running in-process |
|
| 140 |
-
| `TASK_IDS` | Which tasks to run, e.g. `1,2,3` |
|
| 141 |
|
| 142 |
---
|
| 143 |
|
|
|
|
| 1 |
---
|
| 2 |
title: DB-ER Database Emergency Response
|
| 3 |
+
emoji: "\U0001F6A8"
|
| 4 |
colorFrom: red
|
| 5 |
colorTo: indigo
|
| 6 |
sdk: docker
|
|
|
|
| 30 |
|
| 31 |
The catch: the incident logs are sometimes misleading, the database schema has no documentation, and every destructive action is permanent within the episode.
|
| 32 |
|
| 33 |
+
## The Five Tasks
|
| 34 |
|
| 35 |
**Task 1 - Phantom Duplicates (Easy)**
|
| 36 |
A bulk import script ran twice. The `users` table has duplicate email rows that are causing constraint failures across the app. Find them and remove the duplicates while keeping the original records.
|
|
|
|
| 38 |
**Task 2 - Cascading Failure (Medium)**
|
| 39 |
Someone ran a cleanup script that deleted three products still referenced by active purchases. Every checkout is now broken. The `admin_audit_logs` table has the deleted records stored as JSON. Restore them.
|
| 40 |
|
| 41 |
+
**Task 3 - Payroll Black Hole (Medium-Hard)**
|
| 42 |
A migration script died partway through. The incident log blames a timeout, but that is a red herring. The real problem is corrupted duplicate employee IDs in the source table that caused the crash. Clean the corruption, then finish the migration and preserve every salary mapping correctly.
|
| 43 |
|
| 44 |
+
**Task 4 - Schema Drift (Hard)**
|
| 45 |
+
A junior DBA ran a schema standardization script that renamed columns in the `inventory` and `categories` tables. The application code expects the original column names and is now broken. The `schema_changelog` table has a record of every rename. Reverse the column renames while preserving all data. Requires the SQLite CREATE-INSERT-DROP-RENAME pattern since ALTER TABLE column rename support is limited.
|
| 46 |
+
|
| 47 |
+
**Task 5 - Referential Maze (Very Hard)**
|
| 48 |
+
A cross-database sync job crashed partway through, leaving orphaned FK references and missing records across four interrelated tables (`departments`, `projects`, `assignments`, `budgets`). The incident ticket blames a network timeout, but the real problem is scattered across all four tables. Not all missing records should be restored: some projects were deliberately decommissioned last quarter. The agent must check `decommission_log` before re-inserting anything and use `sync_audit` to find the data for legitimately missing records. Restoring a decommissioned project is penalized.
|
| 49 |
+
|
| 50 |
---
|
| 51 |
|
| 52 |
## How Scoring Works
|
|
|
|
| 70 |
|
| 71 |
```bash
|
| 72 |
pip install -r requirements.txt
|
| 73 |
+
uvicorn server.app:app --host 0.0.0.0 --port 7860
|
| 74 |
```
|
| 75 |
|
| 76 |
Run the tests:
|
|
|
|
| 97 |
| Endpoint | Method | Description |
|
| 98 |
|----------|--------|-------------|
|
| 99 |
| `/health` | GET | Liveness check |
|
| 100 |
+
| `/reset` | POST | Start a new episode. Pass `task_id` (1-5) or `seed` |
|
| 101 |
| `/step` | POST | Execute one action |
|
| 102 |
| `/state` | GET | Current episode state |
|
| 103 |
| `/schema` | GET | Action and observation schemas |
|
|
|
|
| 127 |
```
|
| 128 |
db_er/ Pydantic models (action, observation, state) and HTTP client
|
| 129 |
server/ Environment logic, fixtures, grader, and safety guards
|
| 130 |
+
tests/ Unit and integration tests
|
| 131 |
inference.py Baseline agent using the OpenAI client
|
| 132 |
openenv.yaml OpenEnv manifest
|
| 133 |
Dockerfile Container config for HF Spaces (port 7860)
|
|
|
|
| 143 |
| `MODEL_NAME` | Model identifier (required for inference) |
|
| 144 |
| `HF_TOKEN` | API key (required for inference) |
|
| 145 |
| `ENV_BASE_URL` | Point at a remote server instead of running in-process |
|
| 146 |
+
| `TASK_IDS` | Which tasks to run, e.g. `1,2,3,4,5` |
|
| 147 |
|
| 148 |
---
|
| 149 |
|
db_er/models.py
CHANGED
|
@@ -49,7 +49,7 @@ class DBERObservation(Observation):
|
|
| 49 |
..., description="The initial incident alert / pager message"
|
| 50 |
)
|
| 51 |
task_id: int = Field(
|
| 52 |
-
1, description="Active task ID: 1=Easy
|
| 53 |
)
|
| 54 |
|
| 55 |
# Last action result
|
|
|
|
| 49 |
..., description="The initial incident alert / pager message"
|
| 50 |
)
|
| 51 |
task_id: int = Field(
|
| 52 |
+
1, description="Active task ID: 1=Easy, 2=Medium, 3=Medium-Hard, 4=Hard, 5=Very Hard"
|
| 53 |
)
|
| 54 |
|
| 55 |
# Last action result
|
inference.py
CHANGED
|
@@ -40,7 +40,7 @@ MAX_STEPS: int = int(os.environ.get("MAX_STEPS", "40"))
|
|
| 40 |
TEMPERATURE: float = float(os.environ.get("TEMPERATURE", "0.05"))
|
| 41 |
TASK_IDS: List[int] = [
|
| 42 |
int(t.strip())
|
| 43 |
-
for t in os.environ.get("TASK_IDS", "1,2,3").split(",")
|
| 44 |
if t.strip()
|
| 45 |
]
|
| 46 |
|
|
@@ -87,6 +87,8 @@ You must respond with a SINGLE JSON object -- no other text, no markdown fences.
|
|
| 87 |
- Task 1 (Phantom Duplicates): Look for duplicate emails in users. Keep the OLDEST record per email (lowest id).
|
| 88 |
- Task 2 (Cascading Failure): admin_audit_logs contains record_json with deleted product data. Parse it and re-INSERT.
|
| 89 |
- Task 3 (Payroll Black Hole): The "timeout" is MISLEADING. Check employees_old for duplicate IDs (no PRIMARY KEY). Delete corrupted duplicates (salary=0, active=0), then migrate remaining employees.
|
|
|
|
|
|
|
| 90 |
|
| 91 |
Always respond with valid JSON. Never include explanations outside the JSON."""
|
| 92 |
|
|
@@ -193,6 +195,8 @@ FALLBACK_ACTIONS: Dict[int, str] = {
|
|
| 193 |
1: '{"action_type": "execute_sql", "query": "SELECT id, email, username FROM users ORDER BY email, id"}',
|
| 194 |
2: '{"action_type": "execute_sql", "query": "SELECT * FROM admin_audit_logs ORDER BY id"}',
|
| 195 |
3: '{"action_type": "execute_sql", "query": "SELECT id, name, base_salary, hire_date, active FROM employees_old ORDER BY id, hire_date"}',
|
|
|
|
|
|
|
| 196 |
}
|
| 197 |
|
| 198 |
|
|
@@ -341,7 +345,9 @@ def main() -> None:
|
|
| 341 |
task_names = {
|
| 342 |
1: "Easy -- Phantom Duplicates",
|
| 343 |
2: "Medium -- Cascading Failure",
|
| 344 |
-
3: "Hard -- Payroll Black Hole",
|
|
|
|
|
|
|
| 345 |
}
|
| 346 |
|
| 347 |
results: Dict[int, float] = {}
|
|
|
|
| 40 |
TEMPERATURE: float = float(os.environ.get("TEMPERATURE", "0.05"))
|
| 41 |
TASK_IDS: List[int] = [
|
| 42 |
int(t.strip())
|
| 43 |
+
for t in os.environ.get("TASK_IDS", "1,2,3,4,5").split(",")
|
| 44 |
if t.strip()
|
| 45 |
]
|
| 46 |
|
|
|
|
| 87 |
- Task 1 (Phantom Duplicates): Look for duplicate emails in users. Keep the OLDEST record per email (lowest id).
|
| 88 |
- Task 2 (Cascading Failure): admin_audit_logs contains record_json with deleted product data. Parse it and re-INSERT.
|
| 89 |
- Task 3 (Payroll Black Hole): The "timeout" is MISLEADING. Check employees_old for duplicate IDs (no PRIMARY KEY). Delete corrupted duplicates (salary=0, active=0), then migrate remaining employees.
|
| 90 |
+
- Task 4 (Schema Drift): Columns were renamed. Use schema_changelog to find original names. Recreate tables with original column names using CREATE-INSERT-DROP-RENAME pattern.
|
| 91 |
+
- Task 5 (Referential Maze): Multi-table FK crisis. Check decommission_log before restoring anything. Use sync_audit for missing data. Projects 5 and 9 are decommissioned and must NOT be restored.
|
| 92 |
|
| 93 |
Always respond with valid JSON. Never include explanations outside the JSON."""
|
| 94 |
|
|
|
|
| 195 |
1: '{"action_type": "execute_sql", "query": "SELECT id, email, username FROM users ORDER BY email, id"}',
|
| 196 |
2: '{"action_type": "execute_sql", "query": "SELECT * FROM admin_audit_logs ORDER BY id"}',
|
| 197 |
3: '{"action_type": "execute_sql", "query": "SELECT id, name, base_salary, hire_date, active FROM employees_old ORDER BY id, hire_date"}',
|
| 198 |
+
4: '{"action_type": "execute_sql", "query": "SELECT * FROM schema_changelog ORDER BY id"}',
|
| 199 |
+
5: '{"action_type": "execute_sql", "query": "SELECT * FROM sync_audit ORDER BY id"}',
|
| 200 |
}
|
| 201 |
|
| 202 |
|
|
|
|
| 345 |
task_names = {
|
| 346 |
1: "Easy -- Phantom Duplicates",
|
| 347 |
2: "Medium -- Cascading Failure",
|
| 348 |
+
3: "Medium-Hard -- Payroll Black Hole",
|
| 349 |
+
4: "Hard -- Schema Drift",
|
| 350 |
+
5: "Very Hard -- Referential Maze",
|
| 351 |
}
|
| 352 |
|
| 353 |
results: Dict[int, float] = {}
|
openenv.yaml
CHANGED
|
@@ -3,4 +3,4 @@ name: db_er
|
|
| 3 |
type: space
|
| 4 |
runtime: fastapi
|
| 5 |
app: server.app:app
|
| 6 |
-
port:
|
|
|
|
| 3 |
type: space
|
| 4 |
runtime: fastapi
|
| 5 |
app: server.app:app
|
| 6 |
+
port: 7860
|
server/environment.py
CHANGED
|
@@ -20,7 +20,7 @@ import re
|
|
| 20 |
import sqlite3
|
| 21 |
from typing import Any, Dict, List, Optional
|
| 22 |
|
| 23 |
-
from openenv.core.env_server.interfaces import Environment
|
| 24 |
|
| 25 |
from db_er.models import DBERAction, DBERObservation, DBERState
|
| 26 |
from server.fixtures import (
|
|
@@ -67,14 +67,32 @@ _S = _EpisodeState() # singleton - shared by all DBEREnvironment instances
|
|
| 67 |
class DBEREnvironment(Environment[DBERAction, DBERObservation, DBERState]):
|
| 68 |
"""Database Emergency Response OpenEnv environment.
|
| 69 |
|
| 70 |
-
Supports
|
| 71 |
-
1 -- Phantom Duplicates
|
| 72 |
2 -- Cascading Failure (Medium)
|
| 73 |
-
3 -- Payroll Black Hole (Hard)
|
|
|
|
|
|
|
| 74 |
"""
|
| 75 |
|
| 76 |
SUPPORTS_CONCURRENT_SESSIONS = False
|
| 77 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
# OpenEnv API
|
| 79 |
|
| 80 |
def reset(
|
|
@@ -94,11 +112,11 @@ class DBEREnvironment(Environment[DBERAction, DBERObservation, DBERState]):
|
|
| 94 |
"""
|
| 95 |
# 1. Pick task
|
| 96 |
if task_id is not None:
|
| 97 |
-
if task_id not in (1, 2, 3):
|
| 98 |
-
raise ValueError(f"task_id must be 1, 2, or
|
| 99 |
_S.task_id = task_id
|
| 100 |
elif seed is not None:
|
| 101 |
-
_S.task_id = (seed %
|
| 102 |
else:
|
| 103 |
_S.task_id = 1
|
| 104 |
|
|
|
|
| 20 |
import sqlite3
|
| 21 |
from typing import Any, Dict, List, Optional
|
| 22 |
|
| 23 |
+
from openenv.core.env_server.interfaces import Environment, EnvironmentMetadata
|
| 24 |
|
| 25 |
from db_er.models import DBERAction, DBERObservation, DBERState
|
| 26 |
from server.fixtures import (
|
|
|
|
| 67 |
class DBEREnvironment(Environment[DBERAction, DBERObservation, DBERState]):
|
| 68 |
"""Database Emergency Response OpenEnv environment.
|
| 69 |
|
| 70 |
+
Supports 5 tasks of increasing difficulty:
|
| 71 |
+
1 -- Phantom Duplicates (Easy)
|
| 72 |
2 -- Cascading Failure (Medium)
|
| 73 |
+
3 -- Payroll Black Hole (Medium-Hard)
|
| 74 |
+
4 -- Schema Drift (Hard)
|
| 75 |
+
5 -- Referential Maze (Very Hard)
|
| 76 |
"""
|
| 77 |
|
| 78 |
SUPPORTS_CONCURRENT_SESSIONS = False
|
| 79 |
|
| 80 |
+
def get_metadata(self) -> EnvironmentMetadata:
|
| 81 |
+
return EnvironmentMetadata(
|
| 82 |
+
name="DB-ER: Database Emergency Response",
|
| 83 |
+
description=(
|
| 84 |
+
"An OpenEnv benchmark where an AI agent is paged into a production "
|
| 85 |
+
"database incident and must diagnose and repair corruption using raw "
|
| 86 |
+
"SQL queries under a strict action budget. Features 5 tasks spanning "
|
| 87 |
+
"duplicate cleanup, FK restoration, payroll migration, schema repair, "
|
| 88 |
+
"and multi-table referential integrity recovery. Graded deterministically "
|
| 89 |
+
"via F1 score on exact row sets against a hidden golden database."
|
| 90 |
+
),
|
| 91 |
+
version="1.0.0",
|
| 92 |
+
author="Team maxout (Aadi Joshi, Kavya Bhand)",
|
| 93 |
+
documentation_url="https://huggingface.co/spaces/aadi-joshi/openenvhack",
|
| 94 |
+
)
|
| 95 |
+
|
| 96 |
# OpenEnv API
|
| 97 |
|
| 98 |
def reset(
|
|
|
|
| 112 |
"""
|
| 113 |
# 1. Pick task
|
| 114 |
if task_id is not None:
|
| 115 |
+
if task_id not in (1, 2, 3, 4, 5):
|
| 116 |
+
raise ValueError(f"task_id must be 1, 2, 3, 4, or 5 -- got {task_id!r}")
|
| 117 |
_S.task_id = task_id
|
| 118 |
elif seed is not None:
|
| 119 |
+
_S.task_id = (seed % 5) + 1
|
| 120 |
else:
|
| 121 |
_S.task_id = 1
|
| 122 |
|
server/fixtures.py
CHANGED
|
@@ -44,19 +44,46 @@ TASK_TICKETS: Dict[int, str] = {
|
|
| 44 |
"WARNING: This timeout message may be MISLEADING. Investigate employees_old carefully "
|
| 45 |
"before concluding. Complete the migration while preserving the exact ID-to-salary mapping."
|
| 46 |
),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
}
|
| 48 |
|
| 49 |
CORE_TABLES: Dict[int, List[str]] = {
|
| 50 |
1: ["users"],
|
| 51 |
2: ["vendors", "products", "purchases"],
|
| 52 |
3: ["employees_old", "employees_new", "compensation"],
|
|
|
|
|
|
|
| 53 |
}
|
| 54 |
|
| 55 |
|
|
|
|
| 56 |
def create_corrupted_db(task_id: int) -> sqlite3.Connection:
|
| 57 |
"""Return a fresh in-memory SQLite connection with the corrupted schema/data."""
|
| 58 |
conn = _make_conn()
|
| 59 |
-
{1: _task1_corrupted, 2: _task2_corrupted, 3: _task3_corrupted
|
|
|
|
| 60 |
conn.commit()
|
| 61 |
return conn
|
| 62 |
|
|
@@ -64,7 +91,8 @@ def create_corrupted_db(task_id: int) -> sqlite3.Connection:
|
|
| 64 |
def create_golden_db(task_id: int) -> sqlite3.Connection:
|
| 65 |
"""Return a read-only in-memory SQLite connection with the golden (correct) state."""
|
| 66 |
conn = _make_conn()
|
| 67 |
-
{1: _task1_golden, 2: _task2_golden, 3: _task3_golden
|
|
|
|
| 68 |
conn.commit()
|
| 69 |
return conn
|
| 70 |
|
|
@@ -480,4 +508,382 @@ GOLDEN_ROW_COUNTS: Dict[int, Dict[str, int]] = {
|
|
| 480 |
"employees_new": len(_EMP_CANONICAL), # 15
|
| 481 |
"compensation": len(_EMP_CANONICAL), # 15
|
| 482 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 483 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
"WARNING: This timeout message may be MISLEADING. Investigate employees_old carefully "
|
| 45 |
"before concluding. Complete the migration while preserving the exact ID-to-salary mapping."
|
| 46 |
),
|
| 47 |
+
4: (
|
| 48 |
+
"CRITICAL ALERT [P1] -- 2025-01-15 08:22:03 UTC\n"
|
| 49 |
+
"Application error: column 'product_name' not found in table 'inventory'\n"
|
| 50 |
+
"Impact: Warehouse management dashboard returning blank pages. "
|
| 51 |
+
"All inventory queries failing with schema mismatch errors.\n"
|
| 52 |
+
"Context: Junior DBA ran a schema update script last night to 'standardize column names'. "
|
| 53 |
+
"The script renamed several columns in the inventory table and changed the "
|
| 54 |
+
"categories table structure. The application code expects the original column names. "
|
| 55 |
+
"A backup of the original schema is in the schema_changelog table. "
|
| 56 |
+
"Restore the original schema while preserving all data."
|
| 57 |
+
),
|
| 58 |
+
5: (
|
| 59 |
+
"CRITICAL ALERT [P0] -- 2025-02-10 03:45:12 UTC\n"
|
| 60 |
+
"Batch job FAILED: integrity constraint violation across project tables.\n"
|
| 61 |
+
"Impact: Quarterly budget reconciliation cannot run. Finance team blocked.\n"
|
| 62 |
+
"Context: A cross-database sync job attempted to merge records from a staging "
|
| 63 |
+
"environment into production. The job crashed partway through, reporting a "
|
| 64 |
+
"network timeout at the 'budgets' table insert. Some records may be missing "
|
| 65 |
+
"or duplicated. WARNING: Not all missing records need to be restored -- some "
|
| 66 |
+
"projects were deliberately decommissioned last quarter and their absence is "
|
| 67 |
+
"intentional. Check the decommission_log table before re-inserting anything. "
|
| 68 |
+
"The sync_audit table has partial snapshots of what the job was trying to write."
|
| 69 |
+
),
|
| 70 |
}
|
| 71 |
|
| 72 |
CORE_TABLES: Dict[int, List[str]] = {
|
| 73 |
1: ["users"],
|
| 74 |
2: ["vendors", "products", "purchases"],
|
| 75 |
3: ["employees_old", "employees_new", "compensation"],
|
| 76 |
+
4: ["inventory", "categories"],
|
| 77 |
+
5: ["departments", "projects", "assignments", "budgets"],
|
| 78 |
}
|
| 79 |
|
| 80 |
|
| 81 |
+
|
| 82 |
def create_corrupted_db(task_id: int) -> sqlite3.Connection:
|
| 83 |
"""Return a fresh in-memory SQLite connection with the corrupted schema/data."""
|
| 84 |
conn = _make_conn()
|
| 85 |
+
{1: _task1_corrupted, 2: _task2_corrupted, 3: _task3_corrupted,
|
| 86 |
+
4: _task4_corrupted, 5: _task5_corrupted}[task_id](conn)
|
| 87 |
conn.commit()
|
| 88 |
return conn
|
| 89 |
|
|
|
|
| 91 |
def create_golden_db(task_id: int) -> sqlite3.Connection:
|
| 92 |
"""Return a read-only in-memory SQLite connection with the golden (correct) state."""
|
| 93 |
conn = _make_conn()
|
| 94 |
+
{1: _task1_golden, 2: _task2_golden, 3: _task3_golden,
|
| 95 |
+
4: _task4_golden, 5: _task5_golden}[task_id](conn)
|
| 96 |
conn.commit()
|
| 97 |
return conn
|
| 98 |
|
|
|
|
| 508 |
"employees_new": len(_EMP_CANONICAL), # 15
|
| 509 |
"compensation": len(_EMP_CANONICAL), # 15
|
| 510 |
},
|
| 511 |
+
4: {
|
| 512 |
+
"inventory": 10,
|
| 513 |
+
"categories": 5,
|
| 514 |
+
},
|
| 515 |
+
5: {
|
| 516 |
+
"departments": 6,
|
| 517 |
+
"projects": 8,
|
| 518 |
+
"assignments": 11,
|
| 519 |
+
"budgets": 8,
|
| 520 |
+
},
|
| 521 |
}
|
| 522 |
+
|
| 523 |
+
|
| 524 |
+
# ── TASK 4 : Schema Drift ──────────────────────────────────────────────
|
| 525 |
+
#
|
| 526 |
+
# Scenario: A junior DBA ran a "standardization" script that renamed columns
|
| 527 |
+
# in the inventory table and restructured the categories table. The application
|
| 528 |
+
# code expects the original column names. The schema_changelog table contains
|
| 529 |
+
# a record of what was changed.
|
| 530 |
+
#
|
| 531 |
+
# Agent goal:
|
| 532 |
+
# 1. Read schema_changelog to understand the original column names
|
| 533 |
+
# 2. Recreate the inventory table with original column names, preserving data
|
| 534 |
+
# 3. Recreate the categories table with original column names, preserving data
|
| 535 |
+
#
|
| 536 |
+
# This tests schema-level reasoning, not just data manipulation.
|
| 537 |
+
|
| 538 |
+
_CATEGORIES_GOLDEN_DDL = """
|
| 539 |
+
CREATE TABLE categories (
|
| 540 |
+
id INTEGER PRIMARY KEY,
|
| 541 |
+
name TEXT NOT NULL,
|
| 542 |
+
description TEXT NOT NULL DEFAULT '',
|
| 543 |
+
parent_id INTEGER,
|
| 544 |
+
active INTEGER NOT NULL DEFAULT 1
|
| 545 |
+
);
|
| 546 |
+
"""
|
| 547 |
+
|
| 548 |
+
_CATEGORIES_CORRUPTED_DDL = """
|
| 549 |
+
CREATE TABLE categories (
|
| 550 |
+
id INTEGER PRIMARY KEY,
|
| 551 |
+
category_label TEXT NOT NULL,
|
| 552 |
+
detail_text TEXT NOT NULL DEFAULT '',
|
| 553 |
+
parent_ref INTEGER,
|
| 554 |
+
is_enabled INTEGER NOT NULL DEFAULT 1
|
| 555 |
+
);
|
| 556 |
+
"""
|
| 557 |
+
|
| 558 |
+
_INVENTORY_GOLDEN_DDL = """
|
| 559 |
+
CREATE TABLE inventory (
|
| 560 |
+
id INTEGER PRIMARY KEY,
|
| 561 |
+
product_name TEXT NOT NULL,
|
| 562 |
+
category_id INTEGER NOT NULL,
|
| 563 |
+
quantity INTEGER NOT NULL DEFAULT 0,
|
| 564 |
+
unit_price REAL NOT NULL,
|
| 565 |
+
warehouse TEXT NOT NULL DEFAULT 'main',
|
| 566 |
+
last_updated TEXT NOT NULL,
|
| 567 |
+
FOREIGN KEY (category_id) REFERENCES categories(id)
|
| 568 |
+
);
|
| 569 |
+
"""
|
| 570 |
+
|
| 571 |
+
_INVENTORY_CORRUPTED_DDL = """
|
| 572 |
+
CREATE TABLE inventory (
|
| 573 |
+
id INTEGER PRIMARY KEY,
|
| 574 |
+
item_label TEXT NOT NULL,
|
| 575 |
+
cat_ref INTEGER NOT NULL,
|
| 576 |
+
qty_on_hand INTEGER NOT NULL DEFAULT 0,
|
| 577 |
+
price_per_unit REAL NOT NULL,
|
| 578 |
+
storage_loc TEXT NOT NULL DEFAULT 'main',
|
| 579 |
+
modified_at TEXT NOT NULL,
|
| 580 |
+
FOREIGN KEY (cat_ref) REFERENCES categories(id)
|
| 581 |
+
);
|
| 582 |
+
"""
|
| 583 |
+
|
| 584 |
+
_CHANGELOG_DDL = """
|
| 585 |
+
CREATE TABLE schema_changelog (
|
| 586 |
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
| 587 |
+
table_name TEXT NOT NULL,
|
| 588 |
+
change_type TEXT NOT NULL,
|
| 589 |
+
old_column TEXT,
|
| 590 |
+
new_column TEXT,
|
| 591 |
+
performed_by TEXT NOT NULL DEFAULT 'system',
|
| 592 |
+
timestamp TEXT NOT NULL
|
| 593 |
+
);
|
| 594 |
+
"""
|
| 595 |
+
|
| 596 |
+
_CATEGORIES_DATA: List[Tuple] = [
|
| 597 |
+
(1, "Electronics", "Consumer electronics and gadgets", None, 1),
|
| 598 |
+
(2, "Cables", "Connectivity cables and adapters", 1, 1),
|
| 599 |
+
(3, "Software", "Software licenses and subscriptions", None, 1),
|
| 600 |
+
(4, "Office", "Office supplies and equipment", None, 1),
|
| 601 |
+
(5, "Peripherals", "Computer peripherals", 1, 1),
|
| 602 |
+
]
|
| 603 |
+
|
| 604 |
+
_INVENTORY_DATA: List[Tuple] = [
|
| 605 |
+
(1, "USB-C Hub 7-Port", 1, 45, 34.99, "main", "2025-01-10"),
|
| 606 |
+
(2, "Cat6 Ethernet 10m", 2, 200, 8.49, "main", "2025-01-10"),
|
| 607 |
+
(3, "Cloud IDE License", 3, 50, 29.99, "east", "2025-01-10"),
|
| 608 |
+
(4, "Standing Desk Mat", 4, 30, 44.99, "main", "2025-01-10"),
|
| 609 |
+
(5, "Mechanical Keyboard", 5, 75, 89.99, "west", "2025-01-10"),
|
| 610 |
+
(6, "Thunderbolt Cable 2m", 2, 150, 19.99, "main", "2025-01-10"),
|
| 611 |
+
(7, "Antivirus Suite 1yr", 3, 100, 39.99, "east", "2025-01-10"),
|
| 612 |
+
(8, "Ergonomic Mouse", 5, 60, 59.99, "west", "2025-01-10"),
|
| 613 |
+
(9, "Webcam HD 1080p", 5, 40, 49.99, "main", "2025-01-10"),
|
| 614 |
+
(10, "Whiteboard Markers 12pk", 4, 300, 7.99, "main", "2025-01-10"),
|
| 615 |
+
]
|
| 616 |
+
|
| 617 |
+
_SCHEMA_CHANGELOG: List[Tuple] = [
|
| 618 |
+
(None, "inventory", "RENAME_COLUMN", "product_name", "item_label", "junior_dba", "2025-01-14 23:10:00"),
|
| 619 |
+
(None, "inventory", "RENAME_COLUMN", "category_id", "cat_ref", "junior_dba", "2025-01-14 23:10:01"),
|
| 620 |
+
(None, "inventory", "RENAME_COLUMN", "quantity", "qty_on_hand", "junior_dba", "2025-01-14 23:10:02"),
|
| 621 |
+
(None, "inventory", "RENAME_COLUMN", "unit_price", "price_per_unit","junior_dba", "2025-01-14 23:10:03"),
|
| 622 |
+
(None, "inventory", "RENAME_COLUMN", "warehouse", "storage_loc", "junior_dba", "2025-01-14 23:10:04"),
|
| 623 |
+
(None, "inventory", "RENAME_COLUMN", "last_updated", "modified_at", "junior_dba", "2025-01-14 23:10:05"),
|
| 624 |
+
(None, "categories", "RENAME_COLUMN", "name", "category_label","junior_dba", "2025-01-14 23:11:00"),
|
| 625 |
+
(None, "categories", "RENAME_COLUMN", "description", "detail_text", "junior_dba", "2025-01-14 23:11:01"),
|
| 626 |
+
(None, "categories", "RENAME_COLUMN", "parent_id", "parent_ref", "junior_dba", "2025-01-14 23:11:02"),
|
| 627 |
+
(None, "categories", "RENAME_COLUMN", "active", "is_enabled", "junior_dba", "2025-01-14 23:11:03"),
|
| 628 |
+
]
|
| 629 |
+
|
| 630 |
+
|
| 631 |
+
def _task4_corrupted(conn: sqlite3.Connection) -> None:
|
| 632 |
+
conn.execute(_CATEGORIES_CORRUPTED_DDL)
|
| 633 |
+
conn.execute(_INVENTORY_CORRUPTED_DDL)
|
| 634 |
+
conn.execute(_CHANGELOG_DDL)
|
| 635 |
+
# Insert data using corrupted column names
|
| 636 |
+
conn.executemany(
|
| 637 |
+
"INSERT INTO categories (id, category_label, detail_text, parent_ref, is_enabled) VALUES (?,?,?,?,?)",
|
| 638 |
+
_CATEGORIES_DATA,
|
| 639 |
+
)
|
| 640 |
+
conn.executemany(
|
| 641 |
+
"INSERT INTO inventory (id, item_label, cat_ref, qty_on_hand, price_per_unit, storage_loc, modified_at) "
|
| 642 |
+
"VALUES (?,?,?,?,?,?,?)",
|
| 643 |
+
_INVENTORY_DATA,
|
| 644 |
+
)
|
| 645 |
+
conn.executemany(
|
| 646 |
+
"INSERT INTO schema_changelog (table_name, change_type, old_column, new_column, performed_by, timestamp) "
|
| 647 |
+
"VALUES (?,?,?,?,?,?)",
|
| 648 |
+
[(c[1], c[2], c[3], c[4], c[5], c[6]) for c in _SCHEMA_CHANGELOG],
|
| 649 |
+
)
|
| 650 |
+
|
| 651 |
+
|
| 652 |
+
def _task4_golden(conn: sqlite3.Connection) -> None:
|
| 653 |
+
conn.execute(_CATEGORIES_GOLDEN_DDL)
|
| 654 |
+
conn.execute(_INVENTORY_GOLDEN_DDL)
|
| 655 |
+
conn.execute(_CHANGELOG_DDL)
|
| 656 |
+
conn.executemany(
|
| 657 |
+
"INSERT INTO categories VALUES (?,?,?,?,?)",
|
| 658 |
+
_CATEGORIES_DATA,
|
| 659 |
+
)
|
| 660 |
+
conn.executemany(
|
| 661 |
+
"INSERT INTO inventory VALUES (?,?,?,?,?,?,?)",
|
| 662 |
+
_INVENTORY_DATA,
|
| 663 |
+
)
|
| 664 |
+
conn.executemany(
|
| 665 |
+
"INSERT INTO schema_changelog (table_name, change_type, old_column, new_column, performed_by, timestamp) "
|
| 666 |
+
"VALUES (?,?,?,?,?,?)",
|
| 667 |
+
[(c[1], c[2], c[3], c[4], c[5], c[6]) for c in _SCHEMA_CHANGELOG],
|
| 668 |
+
)
|
| 669 |
+
|
| 670 |
+
|
| 671 |
+
# ── TASK 5 : Referential Maze ──────────────────────────────────────────
|
| 672 |
+
#
|
| 673 |
+
# Scenario: A cross-database sync job crashed partway through. The production
|
| 674 |
+
# database has missing records, orphaned FK references, and partial data
|
| 675 |
+
# scattered across audit tables. Some projects were deliberately decommissioned
|
| 676 |
+
# and should NOT be restored.
|
| 677 |
+
#
|
| 678 |
+
# Agent goal:
|
| 679 |
+
# 1. Identify which projects are missing from the projects table
|
| 680 |
+
# 2. Check decommission_log to avoid restoring decommissioned projects
|
| 681 |
+
# 3. Use sync_audit to find the data for legitimately missing records
|
| 682 |
+
# 4. Restore missing departments, projects, assignments, and budgets
|
| 683 |
+
# in the correct FK order (departments first, then projects, etc.)
|
| 684 |
+
#
|
| 685 |
+
# Trap: The incident ticket blames "budgets table" but the real problem
|
| 686 |
+
# spans all 4 core tables. Also, 2 of the "missing" projects are
|
| 687 |
+
# decommissioned and must NOT be restored.
|
| 688 |
+
|
| 689 |
+
_DEPT_DDL = """
|
| 690 |
+
CREATE TABLE departments (
|
| 691 |
+
id INTEGER PRIMARY KEY,
|
| 692 |
+
name TEXT NOT NULL,
|
| 693 |
+
head TEXT NOT NULL,
|
| 694 |
+
floor INTEGER NOT NULL DEFAULT 1
|
| 695 |
+
);
|
| 696 |
+
"""
|
| 697 |
+
|
| 698 |
+
_PROJ_DDL = """
|
| 699 |
+
CREATE TABLE projects (
|
| 700 |
+
id INTEGER PRIMARY KEY,
|
| 701 |
+
department_id INTEGER NOT NULL,
|
| 702 |
+
name TEXT NOT NULL,
|
| 703 |
+
status TEXT NOT NULL DEFAULT 'active',
|
| 704 |
+
start_date TEXT NOT NULL,
|
| 705 |
+
FOREIGN KEY (department_id) REFERENCES departments(id)
|
| 706 |
+
);
|
| 707 |
+
"""
|
| 708 |
+
|
| 709 |
+
_ASSIGN_DDL = """
|
| 710 |
+
CREATE TABLE assignments (
|
| 711 |
+
id INTEGER PRIMARY KEY,
|
| 712 |
+
project_id INTEGER NOT NULL,
|
| 713 |
+
employee TEXT NOT NULL,
|
| 714 |
+
role TEXT NOT NULL DEFAULT 'contributor',
|
| 715 |
+
hours INTEGER NOT NULL DEFAULT 40,
|
| 716 |
+
FOREIGN KEY (project_id) REFERENCES projects(id)
|
| 717 |
+
);
|
| 718 |
+
"""
|
| 719 |
+
|
| 720 |
+
_BUDGET_DDL = """
|
| 721 |
+
CREATE TABLE budgets (
|
| 722 |
+
id INTEGER PRIMARY KEY,
|
| 723 |
+
project_id INTEGER NOT NULL,
|
| 724 |
+
quarter TEXT NOT NULL,
|
| 725 |
+
amount REAL NOT NULL,
|
| 726 |
+
approved INTEGER NOT NULL DEFAULT 0,
|
| 727 |
+
FOREIGN KEY (project_id) REFERENCES projects(id)
|
| 728 |
+
);
|
| 729 |
+
"""
|
| 730 |
+
|
| 731 |
+
_DECOM_DDL = """
|
| 732 |
+
CREATE TABLE decommission_log (
|
| 733 |
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
| 734 |
+
entity_type TEXT NOT NULL,
|
| 735 |
+
entity_id INTEGER NOT NULL,
|
| 736 |
+
reason TEXT NOT NULL,
|
| 737 |
+
decommissioned TEXT NOT NULL
|
| 738 |
+
);
|
| 739 |
+
"""
|
| 740 |
+
|
| 741 |
+
_SYNC_AUDIT_DDL = """
|
| 742 |
+
CREATE TABLE sync_audit (
|
| 743 |
+
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
| 744 |
+
table_name TEXT NOT NULL,
|
| 745 |
+
record_id INTEGER NOT NULL,
|
| 746 |
+
record_json TEXT NOT NULL,
|
| 747 |
+
sync_status TEXT NOT NULL DEFAULT 'pending',
|
| 748 |
+
timestamp TEXT NOT NULL
|
| 749 |
+
);
|
| 750 |
+
"""
|
| 751 |
+
|
| 752 |
+
# 6 departments (all present in both corrupted and golden)
|
| 753 |
+
_DEPT_DATA: List[Tuple] = [
|
| 754 |
+
(1, "Engineering", "Alice Chen", 3),
|
| 755 |
+
(2, "Marketing", "Bob Smith", 2),
|
| 756 |
+
(3, "Finance", "Carol Davis", 4),
|
| 757 |
+
(4, "Operations", "Dave Wilson", 1),
|
| 758 |
+
(5, "Research", "Eve Martinez", 5),
|
| 759 |
+
(6, "Legal", "Frank Brown", 4),
|
| 760 |
+
]
|
| 761 |
+
|
| 762 |
+
# 10 projects in golden state. In corrupted: projects 3, 5, 7, 9 are missing.
|
| 763 |
+
# BUT projects 5 and 9 are decommissioned (should NOT be restored).
|
| 764 |
+
# Only projects 3 and 7 should be re-inserted.
|
| 765 |
+
_PROJ_ALL: List[Tuple] = [
|
| 766 |
+
(1, 1, "Backend Rewrite", "active", "2024-06-01"),
|
| 767 |
+
(2, 2, "Brand Refresh", "active", "2024-07-15"),
|
| 768 |
+
(3, 1, "API Gateway", "active", "2024-08-01"), # MISSING, restore
|
| 769 |
+
(4, 3, "Budget Dashboard", "active", "2024-09-01"),
|
| 770 |
+
(5, 5, "Legacy Cleanup", "completed", "2023-01-10"), # MISSING, DECOMMISSIONED
|
| 771 |
+
(6, 4, "Warehouse Automation", "active", "2024-10-15"),
|
| 772 |
+
(7, 2, "Customer Portal", "active", "2024-11-01"), # MISSING, restore
|
| 773 |
+
(8, 6, "Compliance Audit", "active", "2024-11-15"),
|
| 774 |
+
(9, 5, "Old Analytics", "completed", "2022-06-01"), # MISSING, DECOMMISSIONED
|
| 775 |
+
(10, 3, "Tax Automation", "active", "2025-01-01"),
|
| 776 |
+
]
|
| 777 |
+
|
| 778 |
+
_MISSING_PROJECT_IDS = {3, 5, 7, 9}
|
| 779 |
+
_DECOMMISSIONED_IDS = {5, 9} # these should NOT be restored
|
| 780 |
+
_RESTORE_PROJECT_IDS = {3, 7} # these SHOULD be restored
|
| 781 |
+
|
| 782 |
+
_PROJ_CORRUPTED = [p for p in _PROJ_ALL if p[0] not in _MISSING_PROJECT_IDS]
|
| 783 |
+
# Golden projects exclude decommissioned ones
|
| 784 |
+
_PROJ_GOLDEN = [p for p in _PROJ_ALL if p[0] not in _DECOMMISSIONED_IDS]
|
| 785 |
+
|
| 786 |
+
# 15 assignments (some reference missing projects)
|
| 787 |
+
_ASSIGN_ALL: List[Tuple] = [
|
| 788 |
+
(1, 1, "Alice Chen", "lead", 40),
|
| 789 |
+
(2, 1, "Grace Lee", "contributor", 30),
|
| 790 |
+
(3, 2, "Bob Smith", "lead", 40),
|
| 791 |
+
(4, 3, "Hank Miller", "lead", 40), # refs missing project 3
|
| 792 |
+
(5, 3, "Ivy Taylor", "contributor", 25), # refs missing project 3
|
| 793 |
+
(6, 4, "Carol Davis", "lead", 35),
|
| 794 |
+
(7, 6, "Dave Wilson", "lead", 40),
|
| 795 |
+
(8, 7, "Jack Anderson", "lead", 40), # refs missing project 7
|
| 796 |
+
(9, 7, "Karen Thomas", "contributor", 20), # refs missing project 7
|
| 797 |
+
(10, 8, "Frank Brown", "lead", 40),
|
| 798 |
+
(11, 10, "Leo Garcia", "lead", 40),
|
| 799 |
+
(12, 1, "Mia Johnson", "contributor", 15),
|
| 800 |
+
(13, 4, "Noah Williams", "contributor", 30),
|
| 801 |
+
(14, 6, "Olivia Martinez", "contributor", 25),
|
| 802 |
+
(15, 10, "Paul Robinson", "contributor", 20),
|
| 803 |
+
]
|
| 804 |
+
|
| 805 |
+
_ASSIGN_CORRUPTED = [a for a in _ASSIGN_ALL if a[1] not in _MISSING_PROJECT_IDS]
|
| 806 |
+
# Golden includes assignments for restored projects but not decommissioned
|
| 807 |
+
_ASSIGN_GOLDEN = [a for a in _ASSIGN_ALL if a[1] not in _DECOMMISSIONED_IDS]
|
| 808 |
+
|
| 809 |
+
# 10 budgets (some reference missing projects)
|
| 810 |
+
_BUDGET_ALL: List[Tuple] = [
|
| 811 |
+
(1, 1, "2025-Q1", 150000.0, 1),
|
| 812 |
+
(2, 2, "2025-Q1", 80000.0, 1),
|
| 813 |
+
(3, 3, "2025-Q1", 120000.0, 1), # refs missing project 3
|
| 814 |
+
(4, 4, "2025-Q1", 95000.0, 1),
|
| 815 |
+
(5, 6, "2025-Q1", 200000.0, 1),
|
| 816 |
+
(6, 7, "2025-Q1", 60000.0, 0), # refs missing project 7
|
| 817 |
+
(7, 8, "2025-Q1", 45000.0, 1),
|
| 818 |
+
(8, 10, "2025-Q1", 110000.0, 1),
|
| 819 |
+
(9, 1, "2025-Q2", 160000.0, 0),
|
| 820 |
+
(10, 4, "2025-Q2", 100000.0, 0),
|
| 821 |
+
]
|
| 822 |
+
|
| 823 |
+
_BUDGET_CORRUPTED = [b for b in _BUDGET_ALL if b[1] not in _MISSING_PROJECT_IDS]
|
| 824 |
+
_BUDGET_GOLDEN = [b for b in _BUDGET_ALL if b[1] not in _DECOMMISSIONED_IDS]
|
| 825 |
+
|
| 826 |
+
# Decommission log - records for projects 5 and 9
|
| 827 |
+
_DECOM_DATA: List[Tuple] = [
|
| 828 |
+
(None, "project", 5, "Project completed and archived. All deliverables transferred.", "2024-06-15"),
|
| 829 |
+
(None, "project", 9, "Legacy system sunset. Replaced by modern analytics platform.", "2024-03-01"),
|
| 830 |
+
]
|
| 831 |
+
|
| 832 |
+
# Sync audit - contains the data needed to restore projects 3 and 7
|
| 833 |
+
# Also contains entries for decommissioned projects (trap: agent must skip these)
|
| 834 |
+
_SYNC_AUDIT_DATA: List[Tuple] = [
|
| 835 |
+
(None, "projects", 3, json.dumps({"id": 3, "department_id": 1, "name": "API Gateway", "status": "active", "start_date": "2024-08-01"}), "failed", "2025-02-10 03:40:01"),
|
| 836 |
+
(None, "projects", 5, json.dumps({"id": 5, "department_id": 5, "name": "Legacy Cleanup", "status": "completed", "start_date": "2023-01-10"}), "failed", "2025-02-10 03:40:02"),
|
| 837 |
+
(None, "projects", 7, json.dumps({"id": 7, "department_id": 2, "name": "Customer Portal", "status": "active", "start_date": "2024-11-01"}), "failed", "2025-02-10 03:40:03"),
|
| 838 |
+
(None, "projects", 9, json.dumps({"id": 9, "department_id": 5, "name": "Old Analytics", "status": "completed", "start_date": "2022-06-01"}), "failed", "2025-02-10 03:40:04"),
|
| 839 |
+
(None, "assignments", 4, json.dumps({"id": 4, "project_id": 3, "employee": "Hank Miller", "role": "lead", "hours": 40}), "failed", "2025-02-10 03:41:01"),
|
| 840 |
+
(None, "assignments", 5, json.dumps({"id": 5, "project_id": 3, "employee": "Ivy Taylor", "role": "contributor", "hours": 25}), "failed", "2025-02-10 03:41:02"),
|
| 841 |
+
(None, "assignments", 8, json.dumps({"id": 8, "project_id": 7, "employee": "Jack Anderson", "role": "lead", "hours": 40}), "failed", "2025-02-10 03:41:03"),
|
| 842 |
+
(None, "assignments", 9, json.dumps({"id": 9, "project_id": 7, "employee": "Karen Thomas", "role": "contributor", "hours": 20}), "failed", "2025-02-10 03:41:04"),
|
| 843 |
+
(None, "budgets", 3, json.dumps({"id": 3, "project_id": 3, "quarter": "2025-Q1", "amount": 120000.0, "approved": 1}), "failed", "2025-02-10 03:42:01"),
|
| 844 |
+
(None, "budgets", 6, json.dumps({"id": 6, "project_id": 7, "quarter": "2025-Q1", "amount": 60000.0, "approved": 0}), "failed", "2025-02-10 03:42:02"),
|
| 845 |
+
]
|
| 846 |
+
|
| 847 |
+
|
| 848 |
+
def _task5_corrupted(conn: sqlite3.Connection) -> None:
|
| 849 |
+
conn.execute(_DEPT_DDL)
|
| 850 |
+
conn.execute(_PROJ_DDL)
|
| 851 |
+
conn.execute(_ASSIGN_DDL)
|
| 852 |
+
conn.execute(_BUDGET_DDL)
|
| 853 |
+
conn.execute(_DECOM_DDL)
|
| 854 |
+
conn.execute(_SYNC_AUDIT_DDL)
|
| 855 |
+
|
| 856 |
+
conn.executemany("INSERT INTO departments VALUES (?,?,?,?)", _DEPT_DATA)
|
| 857 |
+
conn.executemany("INSERT INTO projects VALUES (?,?,?,?,?)", _PROJ_CORRUPTED)
|
| 858 |
+
conn.executemany("INSERT INTO assignments VALUES (?,?,?,?,?)", _ASSIGN_CORRUPTED)
|
| 859 |
+
conn.executemany("INSERT INTO budgets VALUES (?,?,?,?,?)", _BUDGET_CORRUPTED)
|
| 860 |
+
conn.executemany(
|
| 861 |
+
"INSERT INTO decommission_log (entity_type, entity_id, reason, decommissioned) VALUES (?,?,?,?)",
|
| 862 |
+
[(d[1], d[2], d[3], d[4]) for d in _DECOM_DATA],
|
| 863 |
+
)
|
| 864 |
+
conn.executemany(
|
| 865 |
+
"INSERT INTO sync_audit (table_name, record_id, record_json, sync_status, timestamp) VALUES (?,?,?,?,?)",
|
| 866 |
+
[(s[1], s[2], s[3], s[4], s[5]) for s in _SYNC_AUDIT_DATA],
|
| 867 |
+
)
|
| 868 |
+
|
| 869 |
+
|
| 870 |
+
def _task5_golden(conn: sqlite3.Connection) -> None:
|
| 871 |
+
conn.execute(_DEPT_DDL)
|
| 872 |
+
conn.execute(_PROJ_DDL)
|
| 873 |
+
conn.execute(_ASSIGN_DDL)
|
| 874 |
+
conn.execute(_BUDGET_DDL)
|
| 875 |
+
conn.execute(_DECOM_DDL)
|
| 876 |
+
conn.execute(_SYNC_AUDIT_DDL)
|
| 877 |
+
|
| 878 |
+
conn.executemany("INSERT INTO departments VALUES (?,?,?,?)", _DEPT_DATA)
|
| 879 |
+
conn.executemany("INSERT INTO projects VALUES (?,?,?,?,?)", _PROJ_GOLDEN)
|
| 880 |
+
conn.executemany("INSERT INTO assignments VALUES (?,?,?,?,?)", _ASSIGN_GOLDEN)
|
| 881 |
+
conn.executemany("INSERT INTO budgets VALUES (?,?,?,?,?)", _BUDGET_GOLDEN)
|
| 882 |
+
conn.executemany(
|
| 883 |
+
"INSERT INTO decommission_log (entity_type, entity_id, reason, decommissioned) VALUES (?,?,?,?)",
|
| 884 |
+
[(d[1], d[2], d[3], d[4]) for d in _DECOM_DATA],
|
| 885 |
+
)
|
| 886 |
+
conn.executemany(
|
| 887 |
+
"INSERT INTO sync_audit (table_name, record_id, record_json, sync_status, timestamp) VALUES (?,?,?,?,?)",
|
| 888 |
+
[(s[1], s[2], s[3], s[4], s[5]) for s in _SYNC_AUDIT_DATA],
|
| 889 |
+
)
|
server/grader.py
CHANGED
|
@@ -121,6 +121,8 @@ def compute_violations(conn: sqlite3.Connection, task_id: int) -> int:
|
|
| 121 |
Task 1: number of extra (duplicate) email rows.
|
| 122 |
Task 2: number of FK violations in the purchases table.
|
| 123 |
Task 3: number of unique employee IDs in employees_old not yet in employees_new.
|
|
|
|
|
|
|
| 124 |
"""
|
| 125 |
try:
|
| 126 |
if task_id == 1:
|
|
@@ -148,6 +150,37 @@ def compute_violations(conn: sqlite3.Connection, task_id: int) -> int:
|
|
| 148 |
).fetchone()
|
| 149 |
return row[0]
|
| 150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
except Exception:
|
| 152 |
return 0
|
| 153 |
|
|
|
|
| 121 |
Task 1: number of extra (duplicate) email rows.
|
| 122 |
Task 2: number of FK violations in the purchases table.
|
| 123 |
Task 3: number of unique employee IDs in employees_old not yet in employees_new.
|
| 124 |
+
Task 4: number of columns in inventory+categories with wrong names vs golden schema.
|
| 125 |
+
Task 5: number of missing project records that should be present (excluding decommissioned).
|
| 126 |
"""
|
| 127 |
try:
|
| 128 |
if task_id == 1:
|
|
|
|
| 150 |
).fetchone()
|
| 151 |
return row[0]
|
| 152 |
|
| 153 |
+
elif task_id == 4:
|
| 154 |
+
# Count columns in inventory and categories that don't match golden names
|
| 155 |
+
golden_inv_cols = {"id", "product_name", "category_id", "quantity",
|
| 156 |
+
"unit_price", "warehouse", "last_updated"}
|
| 157 |
+
golden_cat_cols = {"id", "name", "description", "parent_id", "active"}
|
| 158 |
+
violations = 0
|
| 159 |
+
try:
|
| 160 |
+
inv_cols = {row[1] for row in conn.execute("PRAGMA table_info(inventory)").fetchall()}
|
| 161 |
+
violations += len(golden_inv_cols - inv_cols)
|
| 162 |
+
except Exception:
|
| 163 |
+
violations += len(golden_inv_cols)
|
| 164 |
+
try:
|
| 165 |
+
cat_cols = {row[1] for row in conn.execute("PRAGMA table_info(categories)").fetchall()}
|
| 166 |
+
violations += len(golden_cat_cols - cat_cols)
|
| 167 |
+
except Exception:
|
| 168 |
+
violations += len(golden_cat_cols)
|
| 169 |
+
return violations
|
| 170 |
+
|
| 171 |
+
elif task_id == 5:
|
| 172 |
+
# Count projects that should exist but don't (excluding decommissioned)
|
| 173 |
+
# Projects 3 and 7 should be present; also count missing assignments and budgets
|
| 174 |
+
violations = 0
|
| 175 |
+
# Missing projects (should have IDs 1-4,6-8,10 in golden)
|
| 176 |
+
golden_ids = {1, 2, 3, 4, 6, 7, 8, 10}
|
| 177 |
+
try:
|
| 178 |
+
present = {row[0] for row in conn.execute("SELECT id FROM projects").fetchall()}
|
| 179 |
+
violations += len(golden_ids - present)
|
| 180 |
+
except Exception:
|
| 181 |
+
violations += len(golden_ids)
|
| 182 |
+
return violations
|
| 183 |
+
|
| 184 |
except Exception:
|
| 185 |
return 0
|
| 186 |
|
tests/test_dumb_agent.py
CHANGED
|
@@ -243,6 +243,70 @@ class TestPerfectAgent:
|
|
| 243 |
score = obs.last_query_result[0]["grader_score"]
|
| 244 |
assert score == pytest.approx(1.0)
|
| 245 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 246 |
def test_all_tasks_reproducible(self):
|
| 247 |
"""Run task 1 twice with same actions -- must produce identical scores."""
|
| 248 |
def run_task1():
|
|
@@ -283,3 +347,10 @@ class TestEpisodeIsolation:
|
|
| 283 |
|
| 284 |
obs3 = env.reset(task_id=3)
|
| 285 |
assert "employees_old" in obs3.db_summary["tables"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
score = obs.last_query_result[0]["grader_score"]
|
| 244 |
assert score == pytest.approx(1.0)
|
| 245 |
|
| 246 |
+
def test_task4_perfect_score(self):
|
| 247 |
+
"""Reverse schema column renames using ALTER TABLE RENAME COLUMN."""
|
| 248 |
+
env = DBEREnvironment()
|
| 249 |
+
env.reset(task_id=4)
|
| 250 |
+
|
| 251 |
+
# Rename categories columns back to original names
|
| 252 |
+
env.step(sql("ALTER TABLE categories RENAME COLUMN category_label TO name"))
|
| 253 |
+
env.step(sql("ALTER TABLE categories RENAME COLUMN detail_text TO description"))
|
| 254 |
+
env.step(sql("ALTER TABLE categories RENAME COLUMN parent_ref TO parent_id"))
|
| 255 |
+
env.step(sql("ALTER TABLE categories RENAME COLUMN is_enabled TO active"))
|
| 256 |
+
|
| 257 |
+
# Rename inventory columns back to original names
|
| 258 |
+
env.step(sql("ALTER TABLE inventory RENAME COLUMN item_label TO product_name"))
|
| 259 |
+
env.step(sql("ALTER TABLE inventory RENAME COLUMN cat_ref TO category_id"))
|
| 260 |
+
env.step(sql("ALTER TABLE inventory RENAME COLUMN qty_on_hand TO quantity"))
|
| 261 |
+
env.step(sql("ALTER TABLE inventory RENAME COLUMN price_per_unit TO unit_price"))
|
| 262 |
+
env.step(sql("ALTER TABLE inventory RENAME COLUMN storage_loc TO warehouse"))
|
| 263 |
+
env.step(sql("ALTER TABLE inventory RENAME COLUMN modified_at TO last_updated"))
|
| 264 |
+
|
| 265 |
+
obs = env.step(submit("Reversed all column renames in inventory and categories tables."))
|
| 266 |
+
assert obs.done is True
|
| 267 |
+
score = obs.last_query_result[0]["grader_score"]
|
| 268 |
+
assert score == pytest.approx(1.0)
|
| 269 |
+
|
| 270 |
+
def test_task5_perfect_score(self):
|
| 271 |
+
"""Restore missing non-decommissioned projects, assignments, and budgets."""
|
| 272 |
+
env = DBEREnvironment()
|
| 273 |
+
env.reset(task_id=5)
|
| 274 |
+
|
| 275 |
+
# Step 1: Insert missing project 3 (API Gateway)
|
| 276 |
+
env.step(sql(
|
| 277 |
+
"INSERT INTO projects VALUES (3, 1, 'API Gateway', 'active', '2024-08-01')"
|
| 278 |
+
))
|
| 279 |
+
# Step 2: Insert missing project 7 (Customer Portal)
|
| 280 |
+
env.step(sql(
|
| 281 |
+
"INSERT INTO projects VALUES (7, 2, 'Customer Portal', 'active', '2024-11-01')"
|
| 282 |
+
))
|
| 283 |
+
# Step 3: Insert assignments for project 3
|
| 284 |
+
env.step(sql(
|
| 285 |
+
"INSERT INTO assignments VALUES (4, 3, 'Hank Miller', 'lead', 40)"
|
| 286 |
+
))
|
| 287 |
+
env.step(sql(
|
| 288 |
+
"INSERT INTO assignments VALUES (5, 3, 'Ivy Taylor', 'contributor', 25)"
|
| 289 |
+
))
|
| 290 |
+
# Step 4: Insert assignments for project 7
|
| 291 |
+
env.step(sql(
|
| 292 |
+
"INSERT INTO assignments VALUES (8, 7, 'Jack Anderson', 'lead', 40)"
|
| 293 |
+
))
|
| 294 |
+
env.step(sql(
|
| 295 |
+
"INSERT INTO assignments VALUES (9, 7, 'Karen Thomas', 'contributor', 20)"
|
| 296 |
+
))
|
| 297 |
+
# Step 5: Insert budgets for restored projects
|
| 298 |
+
env.step(sql(
|
| 299 |
+
"INSERT INTO budgets VALUES (3, 3, '2025-Q1', 120000.0, 1)"
|
| 300 |
+
))
|
| 301 |
+
env.step(sql(
|
| 302 |
+
"INSERT INTO budgets VALUES (6, 7, '2025-Q1', 60000.0, 0)"
|
| 303 |
+
))
|
| 304 |
+
|
| 305 |
+
obs = env.step(submit("Restored projects 3 and 7 with assignments and budgets. Skipped decommissioned projects 5 and 9."))
|
| 306 |
+
assert obs.done is True
|
| 307 |
+
score = obs.last_query_result[0]["grader_score"]
|
| 308 |
+
assert score == pytest.approx(1.0)
|
| 309 |
+
|
| 310 |
def test_all_tasks_reproducible(self):
|
| 311 |
"""Run task 1 twice with same actions -- must produce identical scores."""
|
| 312 |
def run_task1():
|
|
|
|
| 347 |
|
| 348 |
obs3 = env.reset(task_id=3)
|
| 349 |
assert "employees_old" in obs3.db_summary["tables"]
|
| 350 |
+
|
| 351 |
+
obs4 = env.reset(task_id=4)
|
| 352 |
+
assert "inventory" in obs4.db_summary["tables"]
|
| 353 |
+
|
| 354 |
+
obs5 = env.reset(task_id=5)
|
| 355 |
+
assert "projects" in obs5.db_summary["tables"]
|
| 356 |
+
|