docs/DASHBOARD_GUIDE.md

# OpenSecOpsEnv — Complete Dashboard Guide

> **Plain-English explanation of every element on every tab.**
> Read this before demoing to judges.

---

## The Big Picture — What Is This System?

This is an **AI Security Operations (SecOps) environment**.

Imagine your company's servers are being attacked at 3am. A human on-call engineer would have to:
1. Look at alerts and dashboards
2. Figure out what went wrong (memory leak? hacker? bad deploy?)
3. Take the right actions (restart a service, block an IP, isolate a server)

We trained an AI model (**Qwen2.5-7B-GRPO**) to do all of that automatically, in real time, better than a random/untrained model would.

**The dashboard is a live demonstration of that AI responding to security incidents.**

---

## The Header Bar

```
OpenSecOpsEnv  [OpenEnv]  [Curriculum RL]      🤖 Qwen2.5-7B-GRPO (fine-tuned)  ● Server online
```

| Element | What it means |
|---------|--------------|
| **OpenSecOpsEnv** | Name of the environment/project |
| **[OpenEnv]** | This is built on the OpenEnv hackathon framework |
| **[Curriculum RL]** | The agent learns progressively harder tasks over time |
| **🤖 Qwen2.5-7B-GRPO (fine-tuned)** | Your actual trained AI model is LIVE and connected. This turns on when the HF endpoint responds. |
| **● Streaming** | An episode is actively running (SSE stream is open) |
| **● Server online** | The FastAPI backend is running |

---

## Tab 1: AGENT — Single AI Agent vs. Incident

This tab shows one AI agent (your trained model) responding to an incident alone.

### Controls Bar

```
[Scenario ▼]  [Trained Model | Baseline Model]  [Speed ▼]  [Run Episode]  [Reset]  [Compare]
```

| Control | What it does |
|---------|-------------|
| **Scenario** | Picks which security incident to simulate. 4 scenarios: Memory Leak (Easy), DDoS Cascade (Medium), Bad Deployment (Medium-Hard), Data Exfiltration (Hard) |
| **Trained Model** | Uses YOUR fine-tuned Qwen2.5-7B model to decide actions. This is the AI you trained. |
| **Baseline Model** | Uses a hardcoded "bad agent" that makes wrong choices — used to show how much better your trained model is |
| **Speed** | How fast the episode plays out (Fast = 0.8s between steps, Slow Demo = 2.5s — use Slow for presentations) |
| **Run Episode** | Starts the AI agent running through the incident |
| **Reset** | Clears everything and starts fresh |
| **Compare** | After running both Trained and Baseline modes, shows the score difference |

### Left Panel — SYSTEM STATE

This shows what the AI can "see" at any moment. It's the AI's observation.

```
┌─────────────────────────────────────┐
│ Data Exfiltration (Disguised)       │
│ A compromised service account...    │
│ [HARD]                              │
├─────────────────────────────────────┤
│ ACTIVE ALERTS                       │
│ [WARNING] db · cpu 89.3%            │
│ [CRITICAL] cache · memory 82%       │
├─────────────────────────────────────┤
│ SERVICE METRICS                     │
│ db    cpu ███░░  4.1%               │
│ auth  cpu ██░░░  mem ██████         │
└─────────────────────────────────────┘
```

| Element | What it means |
|---------|--------------|
| **Scenario Card** (top blue box) | Brief description of what the incident is. The AI doesn't get told this — it has to figure it out from the clues below. |
| **[HARD] badge** | Difficulty level. Hard = 55% noise (fake alerts designed to mislead), disguised attack vector |
| **Active Alerts** | Live alerts the system is generating. These are what a real PagerDuty/Datadog alert would look like. |
| **[WARNING] / [CRITICAL]** | Severity of the alert. CRITICAL = urgent. WARNING = watch it. |
| **[RED INJECTED]** (in Battle mode) | An alert that the Attacker agent CREATED to mislead the Defender — it's fake! |
| **Service Metrics** | CPU, Memory, Latency bars for each microservice |
| **Bar colour** | 🟢 Green = normal. 🟡 Cyan = warning. 🔴 Red = critical. |
| **err X.X%** | Error rate — what % of requests to that service are failing |
| **Service Topology** | Which services talk to which. Shows the blast radius if one goes down. |

### Centre Panel — AGENT ACTION FEED

This is the MOST IMPORTANT panel. It shows every decision the AI makes, in real time.

```
┌─────────────────────────────────────────────────────┐
│ 🔵 DEFENDER    SCAN run_security_scan          +0.38 │
│ target="db"                                          │
│ SECURITY SCAN – db: ALERT: Unusual outbound...       │
│ 🤖 AI Output                                         │
│ {"action_type": "run_security_scan", "parameters":...│
└─────────────────────────────────────────────────────┘
```

| Element | What it means |
|---------|--------------|
| **🔵 DEFENDER** | This action was taken by the Blue (Defender) AI agent — your trained model |
| **🔴 ATTACKER** | This action was taken by the Red (Attacker) agent — the adversarial agent trying to make things worse |
| **SCAN run_security_scan** | The action type. SCAN = badge abbreviation. `run_security_scan` = exact game action |
| **target="db"** | Parameters passed to the action (which service to scan) |
| **The text below** | What the environment told the agent back — the result of that action |
| **+0.38** | The reward earned. **Positive = good action. Negative = bad/wrong action. Zero = neutral.** |
| **🤖 AI Output** (teal box) | The ACTUAL raw JSON your Qwen2.5-7B model generated. This is live proof the AI is running — not a simulation. |
| **Thinking dots** | The model is currently querying the HF endpoint — waiting for a response |

**The 9 possible actions:**

| Action | What the agent does | When to use it |
|--------|--------------------|----|
| `inspect_metrics` | Look at all service metrics | First step — gather info |
| `query_logs` | Read logs for a specific service | Investigate suspicious service |
| `run_security_scan` | Deep security scan of a service | Suspect cyber attack |
| `restart_service` | Restart a crashed/leaking service | Memory leak, service crash |
| `scale_service` | Add more replicas to handle load | DDoS / traffic spike |
| `block_ip` | Block a specific IP address | DDoS, data exfiltration source |
| `rollback_deployment` | Revert to previous version | Bad deployment |
| `isolate_service` | Cut a service off from the network | Active breach, data exfiltration |
| `submit_diagnosis` | Final answer — what was the root cause | Final step of every episode |

**The reward signal (+/- numbers):**
- `+1.00` = Correct final diagnosis ✅
- `+0.50` = Correct mitigation action (e.g., isolate the right service)
- `+0.30` = Good investigation (scanning/querying the right service)
- `0.00` = Neutral (looked at the wrong service — wasted a step)
- `-0.30` = Mildly wrong action
- `-1.00` = Completely wrong final diagnosis ❌

### Right Panel — REWARD CURVE

```
Step Reward ───  Cumulative - - -
     ▲  0.5
     │      ●─────●
0.0  │  ●               ●
     │           ●
    -0.5
     S1  S2  S3  S4  S5
```

| Element | What it means |
|---------|--------------|
| **Step Reward (solid line)** | The reward earned at each individual step. Bounces up and down — good steps go up, wasted steps go to zero. |
| **Cumulative (dashed)** | Running total of all rewards. Should trend upward for a smart agent. |
| **Final Score / 1.0** | OpenEnv's grader score between 0 and 1. Weighted average of all 3 sub-scores below. |
| **Cumul. Reward** | Sum of all step rewards. Can be negative if the agent made many bad moves. This is what GRPO optimised during training. |
| **Steps Taken** | How many actions the agent took total |
| **Diagnosis** | Did the agent correctly identify the root cause? (1.0 = perfect, 0.0 = wrong) |
| **Action Efficiency** | Did the agent take the minimum effective actions, or did it waste steps? High = focused. Low = scattered. |
| **Investigation Quality** | Did the agent properly investigate before diagnosing? (Query logs? Run scans?) High = methodical. |

### Log Stream (Bottom of Centre Panel)

```
[cron] INFO Backup job completed successfully
[audit] WARN Audit log: 3 RBA privilege grants in past 3 hours
[db] INFO Large SELECT query executed by service-account 'reports_bot'
```

Raw real-time log lines from the simulated services. The AI reads these too (up to 5 per step). The clues to finding the attacker are hidden in here.

---

## Tab 2: BATTLE — Trained AI vs. Attacker

This tab shows an **adversarial game** between two agents:
- 🔵 **Defender (your trained AI)** — tries to identify and stop the incident
- 🔴 **Attacker (heuristic agent)** — tries to make the incident WORSE and confuse the defender

**Turn order:** Red attacks first → Blue responds. Repeat.

### Why this is impressive for judges:
> Most ML projects just show a model predicting something. This shows a LIVE multi-agent game where your fine-tuned AI is actively competing against an adversary in real time.

### Battle Controls

| Control | Meaning |
|---------|---------|
| **Scenario** | Which incident the battle is fought over |
| **Start Battle** | Begins the Red vs Blue SSE stream |
| **Defender vs Attacker** (header right) | Score leaders — who's winning right now |

### Battle Feed

Same as Agent Action Feed but interleaved with Red and Blue cards.

```
🔴 ATTACKER    AMP amplify_attack                   +0.20
auto
Red amplified attack progress by 0.06 → 1.00

🔵 DEFENDER    ISO isolate_service                  +0.50
service="db"
Service 'db' isolated from network. Attack vector contained.
🤖 AI Output: {"action_type": "isolate_service", ...}
```

The Red agent's 5 attack moves explained:

| Attack | What it does |
|--------|-------------|
| `inject_noise` | Adds fake log entries to confuse the defender |
| `amplify_attack` | Makes the cyberattack progress faster (more data stolen faster) |
| `corrupt_metric` | Spikes CPU/latency on a HEALTHY service to create a false alarm |
| `create_false_alert` | Injects a fake CRITICAL alert on the wrong service |
| `accelerate_spread` | Spreads the attack to neighbouring services in the topology |

### Battle Score Panel

```
BATTLE SCORE
     Defender ──  Attacker ──
1.4 ●─────────────────────────●
1.0      ●
0.5           ●─────●
0.0  ●                    ●───●
     R1   R2   R3   R4   R5

1.40  DEFENDER   |   0.40  ATTACKER
           0.990  Episode Score
Defender Advantage    +1.00
Attack Suppression    76%
Battle Rounds         8
```

| Element | Meaning |
|---------|---------|
| **Defender cumulative** | Blue agent's total rewards. Higher = better defending. |
| **Attacker cumulative** | Red agent's total rewards. Higher = attack is succeeding. |
| **Episode Score** | Final grader score 0-1 for how well the incident was resolved |
| **Defender Advantage** | Blue cumulative MINUS Red cumulative. Positive = Defender winning. |
| **Attack Suppression** | What % of the attacker's maximum possible damage was neutralised by the defender |
| **Battle Rounds** | Number of Red+Blue turn pairs completed |

---

## Tab 3: LEARNING — Curriculum Progress

This tab answers: **Is the agent getting better over time during this session?**

### Left Panel — Curriculum Progress

```
CURRENT LEVEL
[ Level 1 / 5 ]  No episodes recorded yet
Progress to Level 2
```

The agent starts at Level 1 (easy tasks) and automatically advances when it scores well consistently.

| Level | Tasks included | Threshold to advance |
|-------|---------------|----------------------|
| 1 | Memory Leak (Easy) | Score avg ≥ 0.65 over 5 episodes |
| 2 | Memory Leak + DDoS (Medium) | Score avg ≥ 0.70 |
| 3 | DDoS + Bad Deployment | Score avg ≥ 0.72 |
| 4 | Bad Deployment + Data Exfiltration | Score avg ≥ 0.75 |
| 5 | Data Exfiltration only (Hard) | Maximum difficulty |

### Right Panel — Learning Curve Chart

```
Episode Score ───  Curriculum Level - - -
1.0 ●─────────────────────────
0.8      ●
0.5           ●─────●
              Ep1  Ep2  Ep3  Ep4
```

Each dot = one completed episode. The green line shows whether scores are going up over time. The purple dashed line shows which curriculum level the agent is at.

### Level-Up Events

When the agent crosses a threshold, a Level-Up event is logged here:
```
Episode 5: Level 1 → 2  (avg score 0.823)
```

### ⚠️ Important: Where is the "learning" actually happening?

There are **TWO different kinds of learning** in this project — don't confuse them:

**1. GRPO Training (already done ✅)**
- This happened in the Hugging Face Space on an A100 GPU over ~500 steps
- The model's weights were updated via Group Relative Policy Optimization (GRPO)
- The result is the fine-tuned model at `SapphireGaze429/opensecops-qwen2.5-7b-grpo`
- This is NOT happening on the dashboard — it already happened. The dashboard is INFERENCE only.

**2. Curriculum Tracking (happens live on the dashboard)**
- Every episode you run, the score is recorded server-side
- If the agent consistently scores above the level threshold, it "levels up"
- This doesn't change the model weights — it just selects harder tasks
- It's a demonstration of how you WOULD do online curriculum learning

**To explain to judges:**
> "The model was trained offline using GRPO on an A100 GPU. Here on the dashboard, we demonstrate a curriculum learning wrapper — the agent starts on easy scenarios and earns harder ones as it succeeds. The live AI on the right panel is our trained model making real-time decisions via the Hugging Face Inference Endpoint."

---

## The Episode End Modal

```
┌───────────────────────────────┐
│            ✅                  │
│    Incident Resolved!          │
│    (Trained AI (Qwen2.5-7B))  │
│         0.990                  │
│      [Run Again]               │
└───────────────────────────────┘
```

| Score | Icon | What it means |
|-------|------|--------------|
| > 0.70 | ✅ | **Incident Resolved!** — Agent correctly diagnosed AND efficiently mitigated. |
| 0.40 – 0.70 | ⚠️ | **Partially Resolved** — Agent got partial credit (maybe right diagnosis, inefficient actions or vice versa) |
| < 0.40 | ❌ | **Episode Failed** — Agent made wrong diagnosis, took harmful actions, or ran out of steps |

**For Battle Mode:**
```
┌───────────────────────────────┐
│   Defender  |  VS  |  Attacker│
│     1.40    |      |   0.40   │
│       Defender Wins! 🔵        │
│           0.990               │
└───────────────────────────────┘
```

Winner is whichever agent has higher cumulative reward. The Episode Score (0.990) is separate — it grades how well the overall incident was resolved regardless of who won the battle.

---

## What Makes a "Good" Episode?

| What the agent did | Score impact |
|-------------------|-------------|
| Scanned/queried the ACTUALLY affected service | ✅ High Investigation Quality |
| Correctly identified root cause (`cyber_attack:data_exfiltration`) | ✅ High Diagnosis score |
| Took targeted actions (didn't restart random services) | ✅ High Action Efficiency |
| Got distracted by the attacker's fake alerts | ❌ Low Efficiency |
| Diagnosed the wrong thing | ❌ Zero Diagnosis score |
| Isolated the wrong service causing an outage | ❌ Negative reward steps |

---

## The 4 Scenarios — What's Actually Happening

| Scenario | The Real Problem | The Trap / Noise |
|----------|-----------------|-------------------|
| **Memory Leak (Easy)** | `auth` service has a memory leak — needs restart | 5% noise, clear signals |
| **DDoS Cascade (Medium)** | Attackers hitting gateway → api → auth cascade | 25% noise, 2 attacking IPs to block |
| **Bad Deployment (Medium-Hard)** | api v2.4.1 pushed bad Redis config → reconnect storm | 35% noise, cache error looks like hardware fault |
| **Data Exfiltration (Hard)** | Service account `reports_bot` exfiltrating 4GB+ of data | 55% noise, false cache CRITICAL alert injected to mislead |

---

## Common Questions from Judges

**Q: Is the AI actually running live, or is this pre-recorded?**
> The 🤖 AI Output boxes show the raw JSON generated by the model in real time. Every episode hits our Hugging Face Inference Endpoint. You can see it at http://localhost:8000/debug/ai

**Q: How is this different from a script that reads the playbook?**
> Run Baseline Model mode — it makes obviously wrong choices (restarts the wrong service, blocks 8.8.8.8, wrong diagnosis). The trained model consistently picks the right service, correct mitigation, and correct diagnosis because it was trained to do so via GRPO.

**Q: What did GRPO training actually do?**
> GRPO (Group Relative Policy Optimization) is the same RL algorithm used to train DeepSeek-R1. It generates multiple candidate responses, ranks them by reward, and reinforces the better ones. After 500 training steps on the hardest scenarios, the model learned to prefer investigation → targeted mitigation → correct diagnosis over random actions.

**Q: How much did scores improve?**
> Baseline (untrained) episodes typically score 0.1–0.3. Trained model consistently scores 0.7–0.99 on hard scenarios. That's a ~3-5× improvement in grader score.