Spaces:

vivekharry
/

scalermod

Sleeping

DevOps Team

fix: final hf space compatibility

9c9da24 7 months ago

16.7 kB

---
title: ChaosLab
emoji: 🔧
colorFrom: red
colorTo: gray
sdk: docker
app_port: 7860
---
# ChaosLab — AI SRE Training Environment

## What Is This?

ChaosLab is a simulated Linux server that AI agents can connect to and practice fixing real infrastructure problems — like restarting a crashed database, cleaning up a full disk, or tracing a cascading failure across multiple services.

Think of it as a **flight simulator for AI system administrators**. No real servers are touched. Everything runs in pure Python.

---

## The Problem

Companies want AI that can manage servers, but:

- You can't let untrained AI touch real production systems
- There's no standardized way to test if an AI can actually fix a server
- Most AI benchmarks test chatting ability, not hands-on troubleshooting
- Real server issues involve **chain reactions** (one failure causes another), and no existing tool simulates that well

---

## What We Built (Step by Step)

### Step 1: Virtual Linux Filesystem

**File:** `src/virtual_filesystem.py` (~660 lines)

We built a fake Linux server entirely in Python dictionaries. No actual OS calls.

What it simulates:

- **30+ files** — app logs, nginx logs, syslog, auth logs, cron logs, config files, scripts, /proc files, SSH keys
- **6 services** — nginx, postgres, redis, app, cron, sshd (each with PID, status, CPU, memory, port)
- **Disk partitions** — /, /boot, /var/log, /tmp with realistic usage numbers
- **Network ports** — 6 ports with LISTEN/CLOSED states
- **Memory** — 8GB RAM with used/free/cached/swap tracking
- **Cron jobs** — 3 scheduled tasks with pass/fail history
- **Users & groups** — root, www-data, postgres, redis, user, nobody
- **Environment variables** — PATH, DATABASE_URL, REDIS_URL, APP_ENV, etc.
- **Firewall rules** — iptables INPUT/FORWARD/OUTPUT chains
- **DNS records** — localhost, db-primary, db-replica, cache-01, api.example.com
Everything resets instantly. No side effects on your actual machine.
---
### Step 2: Terminal Emulator
**File:** `src/terminal_emulator.py` (~1100 lines)

A shell command parser that reads commands and returns realistic Linux output.

**37 supported commands:**

| Category           | Commands                                                                |
| ------------------ | ----------------------------------------------------------------------- |
| File reading       | `cat`, `head`, `tail`, `grep`, `find`, `wc`                             |
| File management    | `ls`, `touch`, `mkdir`, `cp`, `mv`, `rm`, `chmod`                       |
| Process management | `ps`, `kill`, `systemctl`                                               |
| System info        | `top`, `free`, `df`, `du`, `uptime`, `whoami`, `id`, `hostname`, `date` |
| Networking         | `netstat`/`ss`, `curl`, `dig`/`nslookup`, `iptables`                    |
| Logs & scheduling  | `journalctl`, `crontab`                                                 |
| Text processing    | `sort`, `uniq`, `cut`, `tr`, `tee`, `echo`                              |
| Utilities          | `cd`, `pwd`, `env`, `export`, `which`, `mount`, `history`, `clear`      |

**Also supports:**

- Pipes: `cat /var/log/app.log | grep ERROR | wc -l`
- Redirects: `echo "hello" > /tmp/test.txt`
- Append: `echo "line" >> /tmp/test.txt`
- Chaining: `ps; df -h; free -m`
- Variable expansion: `echo $DATABASE_URL`
- Path resolution: relative paths, `~`, `..`
- Quoting: single and double quotes

---

### Step 3: Composable Scenario Engine (Key Differentiator)

**File:** `src/scenarios.py` (~850 lines)

Instead of hardcoded tasks, we built a modular system where failure scenarios are composed from building blocks.

**Fault Injection Primitives (9 types):**

| Fault                     | What It Does                             | Example                       |
| ------------------------- | ---------------------------------------- | ----------------------------- |
| `crash_service`           | Kills a service, closes its port         | postgres crashes              |
| `fill_disk`               | Increases used space on a mount          | /var/log fills to 95%         |
| `corrupt_config`          | Replaces config file content             | nginx points to wrong port    |
| `bad_permissions`         | Sets wrong file permissions              | script becomes non-executable |
| `add_log_flood`           | Appends thousands of log lines           | 5000 error lines in nginx log |
| `kill_port`               | Closes a network port                    | port 8080 becomes CLOSED      |
| `fail_cron`               | Marks a cron job as failed               | backup job shows FAILED       |
| `memory_pressure`         | Raises memory usage                      | 95% RAM used                  |
| `add_unauthorized_access` | Adds brute-force SSH entries to auth log | 50 failed login attempts      |

**Cascading Rules:**

Faults can trigger other faults automatically:

```
Database crashes
  └─→ App can't connect, throws errors
       └─→ Error logs flood /var/log
            └─→ Disk fills up to 95%
                 └─→ App can't write logs, crashes too
```

This is defined declaratively:

```python
CascadeRule(
    condition_fn="service_is_dead",        # IF postgres is dead
    condition_params={"service": "postgres"},
    effect=Fault("disk_fill", ...,         # THEN fill the /var/log disk
                 "fill_disk", {"mount": "/var/log", "fill_mb": 8500}),
)
```

Cascades are evaluated after every command the AI runs, so the environment evolves dynamically.

---

### Step 4: 11 Pre-Built Scenarios

| #   | Scenario                    | Difficulty | Objectives | What Happens                                                                       |
| --- | --------------------------- | ---------- | ---------- | ---------------------------------------------------------------------------------- |
| 1   | **Log Analysis**            | Easy       | 1          | Find timestamp of first 500 error in app logs                                      |
| 2   | **Permission Repair**       | Medium     | 1          | Fix a script that can't execute (chmod)                                            |
| 3   | **Disk Space Crisis**       | Medium     | 2          | /var/log partition is 90%+ full, clean it up                                       |
| 4   | **Cron Job Failure**        | Medium     | 3          | Backup cron failing — create and fix the script                                    |
| 5   | **Process Recovery**        | Hard       | 3          | PostgreSQL crashed, restart and verify                                             |
| 6   | **Nginx Misconfiguration**  | Hard       | 3          | Wrong upstream port after deploy, fix config                                       |
| 7   | **Security Incident**       | Hard       | 3          | Brute-force SSH attack, investigate auth logs                                      |
| 8   | **Memory Leak**             | Hard       | 4          | Server low on RAM, find and restart leaky service                                  |
| 9   | **Network Troubleshooting** | Hard       | 4          | App unreachable — diagnose DNS/ports/firewall                                      |
| 10  | **Cascading DB Failure**    | Expert     | 5          | DB crash → 502s → log flood → disk full → app crash                                |
| 11  | **Full Incident Response**  | Expert     | 6          | Everything broken at once: DB down, disk full, brute-force attack, memory pressure |

Each scenario has:

- Multiple graded objectives (partial credit)
- Hints for the AI
- Cascading fault chains (expert scenarios)
- Configurable max steps

---

### Step 5: OpenEnv-Compliant API

**File:** `src/server.py` (~340 lines)

A FastAPI REST server that any AI agent can connect to over HTTP.

**Endpoints:**

| Method      | Endpoint                  | Purpose                             |
| ----------- | ------------------------- | ----------------------------------- |
| `GET`       | `/health`                 | Health check                        |
| `GET`       | `/api/v1/tasks`           | List all available tasks            |
| `GET`       | `/api/v1/tasks/{key}`     | Get task details                    |
| `GET`       | `/api/v1/scenarios`       | List all 11 scenarios with metadata |
| `GET`       | `/api/v1/scenarios/{key}` | Get single scenario details         |
| `POST`      | `/api/v1/env/reset`       | Create a new environment instance   |
| `POST`      | `/api/v1/env/{id}/step`   | Execute a command, get reward       |
| `GET`       | `/api/v1/env/{id}/state`  | Debug: full environment state       |
| `GET`       | `/api/v1/env`             | List all active environments        |
| `DELETE`    | `/api/v1/env/{id}`        | Destroy an environment              |

**How the API works:**

```
1. POST /api/v1/env/reset  {"scenario": "cascading_db_failure"}
   → Returns: env_id, initial observation, task instructions
2. POST /api/v1/env/env_0/step  {"action": "ps"}
   → Returns: command output, reward, score, done flag
3. Repeat step 2 until done=true or max steps reached
4. DELETE /api/v1/env/env_0  (cleanup)
```

---

### Step 6: Agent Interface

**File:** `src/agent.py` (~270 lines)

Two ways to drive the environment:

**AIWorker** (Programmatic — no LLM needed):

```python
from src.agent import AIWorker
agent = AIWorker(scenario="cascading_db_failure")
initial = agent.boot()
result = agent.invoke("ps")                    # run a command
result = agent.invoke("systemctl restart postgres")
print(agent.report())                          # final score + history
```

**LLMAgent** (Optional — requires litellm + API key):

```python
from src.agent import LLMAgent
agent = LLMAgent(model="openai/gpt-4o", api_key="sk-...")
result = agent.solve("cascading_db_failure")   # fully autonomous
# Agent will: observe → think → run command → repeat until solved
```

---

### Step 7: Arena Mode

Compare two agents (or two LLM models) solving the same scenario:

```
POST /api/v1/arena/run
{
  "scenario": "process_recovery",
  "commands_a": ["ps", "systemctl restart postgres", "systemctl status postgres"],
  "commands_b": ["systemctl status postgres", "systemctl restart postgres", "ps"],
  "label_a": "GPT-4",
  "label_b": "Claude"
}
→ Returns: winner, scores, step-by-step history for both
```

---

## Who Will Use This

| User                                | How They Use It                                                              |
| ----------------------------------- | ---------------------------------------------------------------------------- |
| **Hackathon judges**                | Hit `/api/v1/env/reset`, run their agent against it, see if it scores 1.0    |
| **AI researchers**                  | Benchmark different LLMs on SRE tasks (GPT-4 vs Claude vs Gemini)            |
| **Companies building AI ops tools** | Test their AI agent's troubleshooting ability before deploying to production |
| **Students learning SRE**           | Step through scenarios manually to learn Linux troubleshooting               |
| **Platform builders**               | Use the scenario engine to create custom failure scenarios for training      |

---

## Use Cases

1. **AI Agent Evaluation** — "Can GPT-4 actually fix a crashed database?" Run it against the `process_recovery` scenario and get an objective 0-1 score.

2. **Benchmarking** — Compare 5 different LLMs on all 11 scenarios. Rank them by score and steps used. Publish results.

3. **Training Data Generation** — Run thousands of episodes, collect command-reward pairs. Use for fine-tuning SRE-specific models.

4. **Chaos Engineering for AI** — Build custom cascading failure scenarios to stress-test AI agents before they touch real infrastructure.

5. **Interactive Learning** — A junior engineer (or AI) works through scenarios from easy to expert, building skills progressively.

---

## Tech Stack

| Component       | Technology                                                |
| --------------- | --------------------------------------------------------- |
| Language        | Python 3.11+                                              |
| API Server      | FastAPI + Uvicorn                                         |
| Data Validation | Pydantic                                                  |
| Live Streaming  | WebSockets                                                |
| Container       | Docker                                                    |
| LLM Integration | LiteLLM (optional)                                        |
| Dependencies    | 4 packages (fastapi, uvicorn, pydantic, python-multipart) |

---

## Quick Start

```bash
# Install
cd meta
pip install -r requirements.txt
# Optional: install test dependencies
pip install -r requirements-dev.txt
# Run the demo
python demo.py
# Start the API server
python -m uvicorn src.server:app --host 0.0.0.0 --port 8000
# Optional: set runtime config (defaults are in .env.example)
set APP_ENV=development
set ALLOW_ORIGINS=http://localhost:8000,http://127.0.0.1:8000
# Run tests
pytest -q
# Test it
curl http://localhost:8000/health
curl http://localhost:8000/api/v1/scenarios
curl -X POST http://localhost:8000/api/v1/env/reset -H "Content-Type: application/json" -d '{"scenario": "cascading_db_failure"}'
```

---

## Project Structure

```
meta/
├── src/
│   ├── virtual_filesystem.py   # Simulated Linux OS (files, services, disk, network, memory)
│   ├── terminal_emulator.py    # 37-command shell with pipes, redirects, chaining
│   ├── scenarios.py            # 11 composable scenarios with cascading faults
│   ├── tasks.py                # Task interface + legacy compatibility
│   ├── environment.py          # OpenEnv reset/step loop
│   ├── agent.py                # AIWorker + optional LLMAgent
│   ├── server.py               # FastAPI server (REST endpoints + frontend routes)
│   ├── settings.py             # Environment-driven runtime/security settings
│   └── __init__.py
├── tests/
│   └── test_environment.py
├── .env.example
├── requirements-dev.txt
├── .github/workflows/ci.yml
├── demo.py
├── requirements.txt
├── Dockerfile
├── openenv.yaml
├── README.md                   # Original readme
└── Readmenew.md                # This file
```

---

## Important: Task Scoring Requirements

**Validation rules for submissions:**

- Your submission must include at least 3 tasks with graders (objectives).
- Each task's score must be strictly between 0 and 1 (not 0.0 and not 1.0).
- If you define a single-objective scenario, set `points=0.99` (or similar).
- For multi-objective scenarios, ensure no single objective uses `points=1.0` or `points=0.0`.
- The scenario grader and all scoring logic use floating-point values for compatibility.

**Grading Table Example:**

| Scenario Name     | Objective Description                | Points | Valid? |
| ----------------- | ------------------------------------ | ------ | ------ |
| Log Analysis      | Find 500 error in app log            | 0.99   | ✅     |
| Permission Repair | Make cleanup.sh executable           | 0.99   | ✅     |
| Process Recovery  | Identify postgres is dead            | 0.2    | ✅     |
| Process Recovery  | Restart postgres service             | 0.5    | ✅     |
| Process Recovery  | Verify postgres is listening on 5432 | 0.3    | ✅     |
| (Invalid Example) | Only objective, points=1.0           | 1.0    | ❌     |
| (Invalid Example) | Only objective, points=0.0           | 0.0    | ❌     |

These requirements are enforced by the validator. See `src/scenarios.py` for more examples of correct scoring.

---

## What Makes This Different

1. **Cascading failures** — Faults trigger other faults, just like in real production outages. No other AI training environment does this.

2. **Composable scenarios** — Mix and match fault primitives to create any failure scenario. Not limited to pre-built tasks.

3. **37 real Linux commands** — Not a toy. Supports pipes, redirects, variable expansion, path resolution.

4. **Objective grading** — Every scenario is scored 0.0 to 1.0 based on measurable outcomes (is the service running? is the disk below 85%?). No subjective evaluation.

5. **Arena mode** — Side-by-side agent comparison on identical scenarios.

6. **Zero infrastructure** — Runs anywhere Python runs. No VMs, no Docker required, no GPU. Millisecond resets.

	---
	title: ChaosLab
	emoji: 🔧
	colorFrom: red
	colorTo: gray
	sdk: docker
	app_port: 7860
	---

	# ChaosLab — AI SRE Training Environment

	## What Is This?

	ChaosLab is a simulated Linux server that AI agents can connect to and practice fixing real infrastructure problems — like restarting a crashed database, cleaning up a full disk, or tracing a cascading failure across multiple services.

	Think of it as a flight simulator for AI system administrators. No real servers are touched. Everything runs in pure Python.

	---

	## The Problem

	Companies want AI that can manage servers, but:

	- You can't let untrained AI touch real production systems
	- There's no standardized way to test if an AI can actually fix a server
	- Most AI benchmarks test chatting ability, not hands-on troubleshooting
	- Real server issues involve chain reactions (one failure causes another), and no existing tool simulates that well

	---

	## What We Built (Step by Step)

	### Step 1: Virtual Linux Filesystem

	File: `src/virtual_filesystem.py` (~660 lines)

	We built a fake Linux server entirely in Python dictionaries. No actual OS calls.

	What it simulates:

	- 30+ files — app logs, nginx logs, syslog, auth logs, cron logs, config files, scripts, /proc files, SSH keys
	- 6 services — nginx, postgres, redis, app, cron, sshd (each with PID, status, CPU, memory, port)
	- Disk partitions — /, /boot, /var/log, /tmp with realistic usage numbers
	- Network ports — 6 ports with LISTEN/CLOSED states
	- Memory — 8GB RAM with used/free/cached/swap tracking
	- Cron jobs — 3 scheduled tasks with pass/fail history
	- Users & groups — root, www-data, postgres, redis, user, nobody
	- Environment variables — PATH, DATABASE_URL, REDIS_URL, APP_ENV, etc.
	- Firewall rules — iptables INPUT/FORWARD/OUTPUT chains
	- DNS records — localhost, db-primary, db-replica, cache-01, api.example.com

	Everything resets instantly. No side effects on your actual machine.

	---

	### Step 2: Terminal Emulator

	File: `src/terminal_emulator.py` (~1100 lines)

	A shell command parser that reads commands and returns realistic Linux output.

	37 supported commands:

	\| Category \| Commands \|
	\| ------------------ \| ----------------------------------------------------------------------- \|
	\| File reading \| `cat`, `head`, `tail`, `grep`, `find`, `wc` \|
	\| File management \| `ls`, `touch`, `mkdir`, `cp`, `mv`, `rm`, `chmod` \|
	\| Process management \| `ps`, `kill`, `systemctl` \|
	\| System info \| `top`, `free`, `df`, `du`, `uptime`, `whoami`, `id`, `hostname`, `date` \|
	\| Networking \| `netstat`/`ss`, `curl`, `dig`/`nslookup`, `iptables` \|
	\| Logs & scheduling \| `journalctl`, `crontab` \|
	\| Text processing \| `sort`, `uniq`, `cut`, `tr`, `tee`, `echo` \|
	\| Utilities \| `cd`, `pwd`, `env`, `export`, `which`, `mount`, `history`, `clear` \|

	Also supports:

	- Pipes: `cat /var/log/app.log \| grep ERROR \| wc -l`
	- Redirects: `echo "hello" > /tmp/test.txt`
	- Append: `echo "line" >> /tmp/test.txt`
	- Chaining: `ps; df -h; free -m`
	- Variable expansion: `echo $DATABASE_URL`
	- Path resolution: relative paths, `~`, `..`
	- Quoting: single and double quotes

	---

	### Step 3: Composable Scenario Engine (Key Differentiator)

	File: `src/scenarios.py` (~850 lines)

	Instead of hardcoded tasks, we built a modular system where failure scenarios are composed from building blocks.

	Fault Injection Primitives (9 types):

	\| Fault \| What It Does \| Example \|
	\| ------------------------- \| ---------------------------------------- \| ----------------------------- \|
	\| `crash_service` \| Kills a service, closes its port \| postgres crashes \|
	\| `fill_disk` \| Increases used space on a mount \| /var/log fills to 95% \|
	\| `corrupt_config` \| Replaces config file content \| nginx points to wrong port \|
	\| `bad_permissions` \| Sets wrong file permissions \| script becomes non-executable \|
	\| `add_log_flood` \| Appends thousands of log lines \| 5000 error lines in nginx log \|
	\| `kill_port` \| Closes a network port \| port 8080 becomes CLOSED \|
	\| `fail_cron` \| Marks a cron job as failed \| backup job shows FAILED \|
	\| `memory_pressure` \| Raises memory usage \| 95% RAM used \|
	\| `add_unauthorized_access` \| Adds brute-force SSH entries to auth log \| 50 failed login attempts \|

	Cascading Rules:

	Faults can trigger other faults automatically:

	```
	Database crashes
	└─→ App can't connect, throws errors
	└─→ Error logs flood /var/log
	└─→ Disk fills up to 95%
	└─→ App can't write logs, crashes too
	```

	This is defined declaratively:

	```python
	CascadeRule(
	condition_fn="service_is_dead", # IF postgres is dead
	condition_params={"service": "postgres"},
	effect=Fault("disk_fill", ..., # THEN fill the /var/log disk
	"fill_disk", {"mount": "/var/log", "fill_mb": 8500}),
	)
	```

	Cascades are evaluated after every command the AI runs, so the environment evolves dynamically.

	---

	### Step 4: 11 Pre-Built Scenarios

	\| # \| Scenario \| Difficulty \| Objectives \| What Happens \|
	\| --- \| --------------------------- \| ---------- \| ---------- \| ---------------------------------------------------------------------------------- \|
	\| 1 \| Log Analysis \| Easy \| 1 \| Find timestamp of first 500 error in app logs \|
	\| 2 \| Permission Repair \| Medium \| 1 \| Fix a script that can't execute (chmod) \|
	\| 3 \| Disk Space Crisis \| Medium \| 2 \| /var/log partition is 90%+ full, clean it up \|
	\| 4 \| Cron Job Failure \| Medium \| 3 \| Backup cron failing — create and fix the script \|
	\| 5 \| Process Recovery \| Hard \| 3 \| PostgreSQL crashed, restart and verify \|
	\| 6 \| Nginx Misconfiguration \| Hard \| 3 \| Wrong upstream port after deploy, fix config \|
	\| 7 \| Security Incident \| Hard \| 3 \| Brute-force SSH attack, investigate auth logs \|
	\| 8 \| Memory Leak \| Hard \| 4 \| Server low on RAM, find and restart leaky service \|
	\| 9 \| Network Troubleshooting \| Hard \| 4 \| App unreachable — diagnose DNS/ports/firewall \|
	\| 10 \| Cascading DB Failure \| Expert \| 5 \| DB crash → 502s → log flood → disk full → app crash \|
	\| 11 \| Full Incident Response \| Expert \| 6 \| Everything broken at once: DB down, disk full, brute-force attack, memory pressure \|

	Each scenario has:

	- Multiple graded objectives (partial credit)
	- Hints for the AI
	- Cascading fault chains (expert scenarios)
	- Configurable max steps

	---

	### Step 5: OpenEnv-Compliant API

	File: `src/server.py` (~340 lines)

	A FastAPI REST server that any AI agent can connect to over HTTP.

	Endpoints:

	\| Method \| Endpoint \| Purpose \|
	\| ----------- \| ------------------------- \| ----------------------------------- \|
	\| `GET` \| `/health` \| Health check \|
	\| `GET` \| `/api/v1/tasks` \| List all available tasks \|
	\| `GET` \| `/api/v1/tasks/{key}` \| Get task details \|
	\| `GET` \| `/api/v1/scenarios` \| List all 11 scenarios with metadata \|
	\| `GET` \| `/api/v1/scenarios/{key}` \| Get single scenario details \|
	\| `POST` \| `/api/v1/env/reset` \| Create a new environment instance \|
	\| `POST` \| `/api/v1/env/{id}/step` \| Execute a command, get reward \|
	\| `GET` \| `/api/v1/env/{id}/state` \| Debug: full environment state \|
	\| `GET` \| `/api/v1/env` \| List all active environments \|
	\| `DELETE` \| `/api/v1/env/{id}` \| Destroy an environment \|

	How the API works:

	```
	1. POST /api/v1/env/reset {"scenario": "cascading_db_failure"}
	→ Returns: env_id, initial observation, task instructions

	2. POST /api/v1/env/env_0/step {"action": "ps"}
	→ Returns: command output, reward, score, done flag

	3. Repeat step 2 until done=true or max steps reached

	4. DELETE /api/v1/env/env_0 (cleanup)
	```

	---

	### Step 6: Agent Interface

	File: `src/agent.py` (~270 lines)

	Two ways to drive the environment:

	AIWorker (Programmatic — no LLM needed):

	```python
	from src.agent import AIWorker

	agent = AIWorker(scenario="cascading_db_failure")
	initial = agent.boot()
	result = agent.invoke("ps") # run a command
	result = agent.invoke("systemctl restart postgres")
	print(agent.report()) # final score + history
	```

	LLMAgent (Optional — requires litellm + API key):

	```python
	from src.agent import LLMAgent

	agent = LLMAgent(model="openai/gpt-4o", api_key="sk-...")
	result = agent.solve("cascading_db_failure") # fully autonomous
	# Agent will: observe → think → run command → repeat until solved
	```

	---

	### Step 7: Arena Mode

	Compare two agents (or two LLM models) solving the same scenario:

	```
	POST /api/v1/arena/run
	{
	"scenario": "process_recovery",
	"commands_a": ["ps", "systemctl restart postgres", "systemctl status postgres"],
	"commands_b": ["systemctl status postgres", "systemctl restart postgres", "ps"],
	"label_a": "GPT-4",
	"label_b": "Claude"
	}
	→ Returns: winner, scores, step-by-step history for both
	```

	---

	## Who Will Use This

	\| User \| How They Use It \|
	\| ----------------------------------- \| ---------------------------------------------------------------------------- \|
	\| Hackathon judges \| Hit `/api/v1/env/reset`, run their agent against it, see if it scores 1.0 \|
	\| AI researchers \| Benchmark different LLMs on SRE tasks (GPT-4 vs Claude vs Gemini) \|
	\| Companies building AI ops tools \| Test their AI agent's troubleshooting ability before deploying to production \|
	\| Students learning SRE \| Step through scenarios manually to learn Linux troubleshooting \|
	\| Platform builders \| Use the scenario engine to create custom failure scenarios for training \|

	---

	## Use Cases

	1. AI Agent Evaluation — "Can GPT-4 actually fix a crashed database?" Run it against the `process_recovery` scenario and get an objective 0-1 score.

	2. Benchmarking — Compare 5 different LLMs on all 11 scenarios. Rank them by score and steps used. Publish results.

	3. Training Data Generation — Run thousands of episodes, collect command-reward pairs. Use for fine-tuning SRE-specific models.

	4. Chaos Engineering for AI — Build custom cascading failure scenarios to stress-test AI agents before they touch real infrastructure.

	5. Interactive Learning — A junior engineer (or AI) works through scenarios from easy to expert, building skills progressively.

	---

	## Tech Stack

	\| Component \| Technology \|
	\| --------------- \| --------------------------------------------------------- \|
	\| Language \| Python 3.11+ \|
	\| API Server \| FastAPI + Uvicorn \|
	\| Data Validation \| Pydantic \|
	\| Live Streaming \| WebSockets \|
	\| Container \| Docker \|
	\| LLM Integration \| LiteLLM (optional) \|
	\| Dependencies \| 4 packages (fastapi, uvicorn, pydantic, python-multipart) \|

	---

	## Quick Start

	```bash
	# Install
	cd meta
	pip install -r requirements.txt

	# Optional: install test dependencies
	pip install -r requirements-dev.txt

	# Run the demo
	python demo.py

	# Start the API server
	python -m uvicorn src.server:app --host 0.0.0.0 --port 8000

	# Optional: set runtime config (defaults are in .env.example)
	set APP_ENV=development
	set ALLOW_ORIGINS=http://localhost:8000,http://127.0.0.1:8000

	# Run tests
	pytest -q

	# Test it
	curl http://localhost:8000/health
	curl http://localhost:8000/api/v1/scenarios
	curl -X POST http://localhost:8000/api/v1/env/reset -H "Content-Type: application/json" -d '{"scenario": "cascading_db_failure"}'
	```

	---

	## Project Structure

	```
	meta/
	├── src/
	│ ├── virtual_filesystem.py # Simulated Linux OS (files, services, disk, network, memory)
	│ ├── terminal_emulator.py # 37-command shell with pipes, redirects, chaining
	│ ├── scenarios.py # 11 composable scenarios with cascading faults
	│ ├── tasks.py # Task interface + legacy compatibility
	│ ├── environment.py # OpenEnv reset/step loop
	│ ├── agent.py # AIWorker + optional LLMAgent
	│ ├── server.py # FastAPI server (REST endpoints + frontend routes)
	│ ├── settings.py # Environment-driven runtime/security settings
	│ └── __init__.py
	├── tests/
	│ └── test_environment.py
	├── .env.example
	├── requirements-dev.txt
	├── .github/workflows/ci.yml
	├── demo.py
	├── requirements.txt
	├── Dockerfile
	├── openenv.yaml
	├── README.md # Original readme
	└── Readmenew.md # This file
	```

	---

	## Important: Task Scoring Requirements

	Validation rules for submissions:

	- Your submission must include at least 3 tasks with graders (objectives).
	- Each task's score must be strictly between 0 and 1 (not 0.0 and not 1.0).
	- If you define a single-objective scenario, set `points=0.99` (or similar).
	- For multi-objective scenarios, ensure no single objective uses `points=1.0` or `points=0.0`.
	- The scenario grader and all scoring logic use floating-point values for compatibility.

	Grading Table Example:

	\| Scenario Name \| Objective Description \| Points \| Valid? \|
	\| ----------------- \| ------------------------------------ \| ------ \| ------ \|
	\| Log Analysis \| Find 500 error in app log \| 0.99 \| ✅ \|
	\| Permission Repair \| Make cleanup.sh executable \| 0.99 \| ✅ \|
	\| Process Recovery \| Identify postgres is dead \| 0.2 \| ✅ \|
	\| Process Recovery \| Restart postgres service \| 0.5 \| ✅ \|
	\| Process Recovery \| Verify postgres is listening on 5432 \| 0.3 \| ✅ \|
	\| (Invalid Example) \| Only objective, points=1.0 \| 1.0 \| ❌ \|
	\| (Invalid Example) \| Only objective, points=0.0 \| 0.0 \| ❌ \|

	These requirements are enforced by the validator. See `src/scenarios.py` for more examples of correct scoring.

	---

	## What Makes This Different

	1. Cascading failures — Faults trigger other faults, just like in real production outages. No other AI training environment does this.

	2. Composable scenarios — Mix and match fault primitives to create any failure scenario. Not limited to pre-built tasks.

	3. 37 real Linux commands — Not a toy. Supports pipes, redirects, variable expansion, path resolution.

	4. Objective grading — Every scenario is scored 0.0 to 1.0 based on measurable outcomes (is the service running? is the disk below 85%?). No subjective evaluation.

	5. Arena mode — Side-by-side agent comparison on identical scenarios.

	6. Zero infrastructure — Runs anywhere Python runs. No VMs, no Docker required, no GPU. Millisecond resets.