README.md

metadata

title: ChaosLab
emoji: 🔧
colorFrom: red
colorTo: gray
sdk: docker
app_port: 7860

ChaosLab — AI SRE Training Environment

What Is This?

ChaosLab is a simulated Linux server that AI agents can connect to and practice fixing real infrastructure problems — like restarting a crashed database, cleaning up a full disk, or tracing a cascading failure across multiple services.

Think of it as a flight simulator for AI system administrators. No real servers are touched. Everything runs in pure Python.

---

The Problem

Companies want AI that can manage servers, but:

• You can't let untrained AI touch real production systems
• There's no standardized way to test if an AI can actually fix a server
• Most AI benchmarks test chatting ability, not hands-on troubleshooting
• Real server issues involve chain reactions (one failure causes another), and no existing tool simulates that well

---

What We Built (Step by Step)

Step 1: Virtual Linux Filesystem

File: src/virtual_filesystem.py (~660 lines)

We built a fake Linux server entirely in Python dictionaries. No actual OS calls.

What it simulates:

• 30+ files — app logs, nginx logs, syslog, auth logs, cron logs, config files, scripts, /proc files, SSH keys
• 6 services — nginx, postgres, redis, app, cron, sshd (each with PID, status, CPU, memory, port)
• Disk partitions — /, /boot, /var/log, /tmp with realistic usage numbers
• Network ports — 6 ports with LISTEN/CLOSED states
• Memory — 8GB RAM with used/free/cached/swap tracking
• Cron jobs — 3 scheduled tasks with pass/fail history
• Users & groups — root, www-data, postgres, redis, user, nobody
• Environment variables — PATH, DATABASE_URL, REDIS_URL, APP_ENV, etc.
• Firewall rules — iptables INPUT/FORWARD/OUTPUT chains
• DNS records — localhost, db-primary, db-replica, cache-01, api.example.com

Everything resets instantly. No side effects on your actual machine.

---

Step 2: Terminal Emulator

File: src/terminal_emulator.py (~1100 lines)

A shell command parser that reads commands and returns realistic Linux output.

37 supported commands:

Category	Commands
File reading	cat, head, tail, grep, find, wc
File management	ls, touch, mkdir, cp, mv, rm, chmod
Process management	ps, kill, systemctl
System info	top, free, df, du, uptime, whoami, id, hostname, date
Networking	netstat/ss, curl, dig/nslookup, iptables
Logs & scheduling	journalctl, crontab
Text processing	sort, uniq, cut, tr, tee, echo
Utilities	cd, pwd, env, export, which, mount, history, clear

Also supports:

• Pipes: cat /var/log/app.log | grep ERROR | wc -l
• Redirects: echo "hello" > /tmp/test.txt
• Append: echo "line" >> /tmp/test.txt
• Chaining: ps; df -h; free -m
• Variable expansion: echo $DATABASE_URL
• Path resolution: relative paths, ~, ..
• Quoting: single and double quotes

---

Step 3: Composable Scenario Engine (Key Differentiator)

File: src/scenarios.py (~850 lines)

Instead of hardcoded tasks, we built a modular system where failure scenarios are composed from building blocks.

Fault Injection Primitives (9 types):

Fault	What It Does	Example
crash_service	Kills a service, closes its port	postgres crashes
fill_disk	Increases used space on a mount	/var/log fills to 95%
corrupt_config	Replaces config file content	nginx points to wrong port
bad_permissions	Sets wrong file permissions	script becomes non-executable
add_log_flood	Appends thousands of log lines	5000 error lines in nginx log
kill_port	Closes a network port	port 8080 becomes CLOSED
fail_cron	Marks a cron job as failed	backup job shows FAILED
memory_pressure	Raises memory usage	95% RAM used
add_unauthorized_access	Adds brute-force SSH entries to auth log	50 failed login attempts

Cascading Rules:

Faults can trigger other faults automatically:

Database crashes
  └─→ App can't connect, throws errors
       └─→ Error logs flood /var/log
            └─→ Disk fills up to 95%
                 └─→ App can't write logs, crashes too

This is defined declaratively:

CascadeRule(
    condition_fn="service_is_dead",        # IF postgres is dead
    condition_params={"service": "postgres"},
    effect=Fault("disk_fill", ...,         # THEN fill the /var/log disk
                 "fill_disk", {"mount": "/var/log", "fill_mb": 8500}),
)

Cascades are evaluated after every command the AI runs, so the environment evolves dynamically.

---

Step 4: 11 Pre-Built Scenarios

#	Scenario	Difficulty	Objectives	What Happens
1	Log Analysis	Easy	1	Find timestamp of first 500 error in app logs
2	Permission Repair	Medium	1	Fix a script that can't execute (chmod)
3	Disk Space Crisis	Medium	2	/var/log partition is 90%+ full, clean it up
4	Cron Job Failure	Medium	3	Backup cron failing — create and fix the script
5	Process Recovery	Hard	3	PostgreSQL crashed, restart and verify
6	Nginx Misconfiguration	Hard	3	Wrong upstream port after deploy, fix config
7	Security Incident	Hard	3	Brute-force SSH attack, investigate auth logs
8	Memory Leak	Hard	4	Server low on RAM, find and restart leaky service
9	Network Troubleshooting	Hard	4	App unreachable — diagnose DNS/ports/firewall
10	Cascading DB Failure	Expert	5	DB crash → 502s → log flood → disk full → app crash
11	Full Incident Response	Expert	6	Everything broken at once: DB down, disk full, brute-force attack, memory pressure

Each scenario has:

• Multiple graded objectives (partial credit)
• Hints for the AI
• Cascading fault chains (expert scenarios)
• Configurable max steps

---

Step 5: OpenEnv-Compliant API

File: src/server.py (~340 lines)

A FastAPI REST server that any AI agent can connect to over HTTP.

Endpoints:

Method	Endpoint	Purpose
GET	/health	Health check
GET	/api/v1/tasks	List all available tasks
GET	/api/v1/tasks/{key}	Get task details
GET	/api/v1/scenarios	List all 11 scenarios with metadata
GET	/api/v1/scenarios/{key}	Get single scenario details
POST	/api/v1/env/reset	Create a new environment instance
POST	/api/v1/env/{id}/step	Execute a command, get reward
GET	/api/v1/env/{id}/state	Debug: full environment state
GET	/api/v1/env	List all active environments
DELETE	/api/v1/env/{id}	Destroy an environment

How the API works:

1. POST /api/v1/env/reset  {"scenario": "cascading_db_failure"}
   → Returns: env_id, initial observation, task instructions

2. POST /api/v1/env/env_0/step  {"action": "ps"}
   → Returns: command output, reward, score, done flag

3. Repeat step 2 until done=true or max steps reached

4. DELETE /api/v1/env/env_0  (cleanup)

---

Step 6: Agent Interface

File: src/agent.py (~270 lines)

Two ways to drive the environment:

AIWorker (Programmatic — no LLM needed):

from src.agent import AIWorker

agent = AIWorker(scenario="cascading_db_failure")
initial = agent.boot()
result = agent.invoke("ps")                    # run a command
result = agent.invoke("systemctl restart postgres")
print(agent.report())                          # final score + history

LLMAgent (Optional — requires litellm + API key):

from src.agent import LLMAgent

agent = LLMAgent(model="openai/gpt-4o", api_key="sk-...")
result = agent.solve("cascading_db_failure")   # fully autonomous
# Agent will: observe → think → run command → repeat until solved

---

Step 7: Arena Mode

Compare two agents (or two LLM models) solving the same scenario:

POST /api/v1/arena/run
{
  "scenario": "process_recovery",
  "commands_a": ["ps", "systemctl restart postgres", "systemctl status postgres"],
  "commands_b": ["systemctl status postgres", "systemctl restart postgres", "ps"],
  "label_a": "GPT-4",
  "label_b": "Claude"
}
→ Returns: winner, scores, step-by-step history for both

---

Who Will Use This

User	How They Use It
Hackathon judges	Hit /api/v1/env/reset, run their agent against it, see if it scores 1.0
AI researchers	Benchmark different LLMs on SRE tasks (GPT-4 vs Claude vs Gemini)
Companies building AI ops tools	Test their AI agent's troubleshooting ability before deploying to production
Students learning SRE	Step through scenarios manually to learn Linux troubleshooting
Platform builders	Use the scenario engine to create custom failure scenarios for training

---

Use Cases

1. AI Agent Evaluation — "Can GPT-4 actually fix a crashed database?" Run it against the process_recovery scenario and get an objective 0-1 score.

2. Benchmarking — Compare 5 different LLMs on all 11 scenarios. Rank them by score and steps used. Publish results.

3. Training Data Generation — Run thousands of episodes, collect command-reward pairs. Use for fine-tuning SRE-specific models.

4. Chaos Engineering for AI — Build custom cascading failure scenarios to stress-test AI agents before they touch real infrastructure.

5. Interactive Learning — A junior engineer (or AI) works through scenarios from easy to expert, building skills progressively.

---

Tech Stack

Component	Technology
Language	Python 3.11+
API Server	FastAPI + Uvicorn
Data Validation	Pydantic
Live Streaming	WebSockets
Container	Docker
LLM Integration	LiteLLM (optional)
Dependencies	4 packages (fastapi, uvicorn, pydantic, python-multipart)

---

Quick Start

# Install
cd meta
pip install -r requirements.txt

# Optional: install test dependencies
pip install -r requirements-dev.txt

# Run the demo
python demo.py

# Start the API server
python -m uvicorn src.server:app --host 0.0.0.0 --port 8000

# Optional: set runtime config (defaults are in .env.example)
set APP_ENV=development
set ALLOW_ORIGINS=http://localhost:8000,http://127.0.0.1:8000

# Run tests
pytest -q

# Test it
curl http://localhost:8000/health
curl http://localhost:8000/api/v1/scenarios
curl -X POST http://localhost:8000/api/v1/env/reset -H "Content-Type: application/json" -d '{"scenario": "cascading_db_failure"}'

---

Project Structure

meta/
├── src/
│   ├── virtual_filesystem.py   # Simulated Linux OS (files, services, disk, network, memory)
│   ├── terminal_emulator.py    # 37-command shell with pipes, redirects, chaining
│   ├── scenarios.py            # 11 composable scenarios with cascading faults
│   ├── tasks.py                # Task interface + legacy compatibility
│   ├── environment.py          # OpenEnv reset/step loop
│   ├── agent.py                # AIWorker + optional LLMAgent
│   ├── server.py               # FastAPI server (REST endpoints + frontend routes)
│   ├── settings.py             # Environment-driven runtime/security settings
│   └── __init__.py
├── tests/
│   └── test_environment.py
├── .env.example
├── requirements-dev.txt
├── .github/workflows/ci.yml
├── demo.py
├── requirements.txt
├── Dockerfile
├── openenv.yaml
├── README.md                   # Original readme
└── Readmenew.md                # This file

---

Important: Task Scoring Requirements

Validation rules for submissions:

• Your submission must include at least 3 tasks with graders (objectives).
• Each task's score must be strictly between 0 and 1 (not 0.0 and not 1.0).
• If you define a single-objective scenario, set points=0.99 (or similar).
• For multi-objective scenarios, ensure no single objective uses points=1.0 or points=0.0.
• The scenario grader and all scoring logic use floating-point values for compatibility.

Grading Table Example:

Scenario Name	Objective Description	Points	Valid?
Log Analysis	Find 500 error in app log	0.99	✅
Permission Repair	Make cleanup.sh executable	0.99	✅
Process Recovery	Identify postgres is dead	0.2	✅
Process Recovery	Restart postgres service	0.5	✅
Process Recovery	Verify postgres is listening on 5432	0.3	✅
(Invalid Example)	Only objective, points=1.0	1.0	❌
(Invalid Example)	Only objective, points=0.0	0.0	❌

These requirements are enforced by the validator. See src/scenarios.py for more examples of correct scoring.

---

What Makes This Different

1. Cascading failures — Faults trigger other faults, just like in real production outages. No other AI training environment does this.

2. Composable scenarios — Mix and match fault primitives to create any failure scenario. Not limited to pre-built tasks.

3. 37 real Linux commands — Not a toy. Supports pipes, redirects, variable expansion, path resolution.

4. Objective grading — Every scenario is scored 0.0 to 1.0 based on measurable outcomes (is the service running? is the disk below 85%?). No subjective evaluation.

5. Arena mode — Side-by-side agent comparison on identical scenarios.

6. Zero infrastructure — Runs anywhere Python runs. No VMs, no Docker required, no GPU. Millisecond resets.