Readme updated82

README.md

@@ -1,96 +1,160 @@
-import os
-import sys
-import torch
-import matplotlib.pyplot as plt
-from datasets import load_dataset
-from trl import GRPOTrainer, GRPOConfig
-from unsloth import FastLanguageModel, PatchFastRL
-
-# ── 1. PATCH & ENVIRONMENT SETUP ──
-PatchFastRL("GRPO", FastLanguageModel)
-
-# Force Colab to see your disaster_grid package
-repo_path = "/content/repo"
-if repo_path not in sys.path:
-    sys.path.insert(0, repo_path)
-
-# Manual imports to bypass standard ModuleNotFound issues
-from src.disaster_grid.environment import CityGrid
-from src.disaster_grid.rewards import compute_reward
-
-# ── 2. LOAD MODEL & TOKENIZER ──
-model, tokenizer = FastLanguageModel.from_pretrained(
-    model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit",
-    max_seq_length = 1024,
-    load_in_4bit = True,
-)
+---
+title: AI-Kinetic Disaster Recovery Grid
+emoji: 🏙️
+colorFrom: red
+colorTo: yellow
+sdk: docker
+app_file: app.py
+pinned: true
+license: mit
+short_description: Multi-Agent RL crisis-grid recovery on OpenEnv
+tags:
+  - reinforcement-learning
+  - multi-agent
+  - openenv
+  - pytorch
+  - llm-agents
+  - hackathon
+---
 
-# ── 3. DATASET PREPARATION ──
-# Loading the small synthetic dataset for rapid alignment
-dataset = load_dataset("json", data_files=f"{repo_path}/train/synthetic_data.json", split="train")
-
-# ── 4. REWARD FUNCTION WRAPPER ──
-env = CityGrid()
-
-def disaster_reward_func(prompts, completions, **kwargs):
-    """Executes model actions in the physics engine to calculate R1/R2/R3."""
-    rewards = []
-    for prompt, completion in zip(prompts, completions):
-        env.reset()
-        # Extract content from assistant response
-        content = completion[0]["content"] if isinstance(completion, list) else completion
-        
-        # Step through environment and get the verifier receipt
-        obs, reward_env, done, truncated, info = env.step(content)
-        
-        # Calculate final weighted score
-        score = compute_reward(info)
-        rewards.append(score)
-    return rewards
-
-# ── 5. TRAINER CONFIGURATION ──
-training_args = GRPOConfig(
-    output_dir = "grpo_results",
-    learning_rate = 2e-5,    
-    logging_steps = 5,       
-    max_steps = 100,
-    per_device_train_batch_size = 2,
-    gradient_accumulation_steps = 2,
-    max_completion_length = 128,
-    report_to = "none", # Faster without tracking
-)
+# 🏙️ AI-Kinetic Disaster Recovery Grid
+
+> A Multi-Agent Reinforcement Learning environment built on **OpenEnv** for the **Meta PyTorch × Scaler OpenEnv Hackathon**.
+
+---
+
+## 🚨 Problem Statement
+
+Modern Large Language Models excel at conversational tasks and static reasoning, but they fail catastrophically at **real-time resource management** and **long-horizon planning** during active crises.
+
+When a disaster strikes — a solar flare, cyber-attack, or severe weather event — infrastructure grids suffer cascading failures. In these scenarios:
+
+- **Information is sparse.** Communication lines sever, creating partial observability.
+- **Resources are finite.** Repairing one sector may mean abandoning another.
+- **Rewards are delayed.** Actions taken now (gathering tools) may not yield results until much later.
+
+Traditional centralized AI models assume perfect information and unlimited bandwidth. Placed in a constrained, decaying environment, they collapse.
+
+---
+
+## 💡 The Proposed Solution
+
+We built the **AI-Kinetic Disaster Recovery Grid** — a Multi-Agent Reinforcement Learning (MARL) environment. Instead of a single "Omni-Agent," we deployed a distributed team of specialized agents acting on a dynamic 5×5 city grid.
+
+### Agent Roles
+
+| Agent | Role |
+|---|---|
+| 🛠️ **Scavenger** | Navigates the grid to collect scarce battery packs and repair parts. |
+| ⚙️ **Engineer** | Executes complex, multi-step repair sequences. |
+| 📡 **Dispatcher** | Holds the global map and intelligently guides the field agents. |
+
+---
+
+## ⚙️ Workflow
+
+### 1. Environment Simulation (OpenEnv)
+
+The world is modeled as a 5×5 matrix using `openenv-core`. Each sector tracks internal state: **Health**, **Power**, and **Risk**.
+
+![OpenEnv Local Development Configuration](./assets/environment_config.jpeg)
+*VS Code showing `environment.py` defining the `CityGrid` class and episode dynamics.*
+
+### 2. Training Scripts (GRPO + Unsloth)
+
+We used **Group Relative Policy Optimization (GRPO)**, accelerated with **Unsloth** to manage memory constraints during training.
+
+```python
+from openenv import OpenEnvGrid
+from trl import GRPOTrainer
+
+env = OpenEnvGrid(size=5, agents=["scavenger", "engineer", "dispatcher"])
 
 trainer = GRPOTrainer(
-    model = model,
-    reward_funcs = [disaster_reward_func],
-    args = training_args,
-    train_dataset = dataset,
+    model="meta-llama/Llama-3-8B",
+    args=config,
+    train_dataset=env.get_training_data(),
 )
+trainer.train()
+```
+
+---
+
+## 🧪 Validation
+
+### Local Validation & Execution
+
+Before pushing to the Hugging Face Space, we validate agent logic and entropy sequences via a local CLI built on `openenv-core` for rapid debugging of the 5×5 grid state transitions.
+
+<video controls src="./assets/validation_video.mp4" width="100%"></video>
+
+![Local Terminal Simulation Output](./assets/terminal_output.jpeg)
+*Local terminal simulator showing a completed 50-step sequence tracking City Health and Energy metrics.*
+
+### Live Agent Simulation (Operational Theatre)
+
+To visualize multi-agent orchestration, we built a web-based UI — the **Operational Theatre** — deployed on Hugging Face Spaces. It allows random anomaly injection while observing LLM-powered agents respond in real time.
+
+<video controls src="./assets/operational_theatre.mp4" width="100%"></video>
+*HF Space UI showing the autonomous agent navigating the grid with the Agent Brain Log streaming reasoning.*
+
+---
+
+## 🌍 Real-World Applications
+
+- **Autonomous Smart Grid Management** — Dynamically rerouting power around physical failures to prevent cascading blackouts.
+- **Search and Rescue Drone Swarms** — Coordinating ground and air drones in disaster zones with limited battery life.
+- **Self-Healing Server Networks** — Autonomously rerouting global traffic and deploying patches during cyber-attacks.
+
+---
+
+## ⚖️ Advantages & Trade-offs
+
+### ✅ Advantages
+
+- **Task Specialization** — Specialized roles converge faster than a single omni-agent.
+- **Fault Tolerance** — No single point of failure; the Dispatcher reroutes around incapacitated agents.
+- **Realistic Observability** — Native "fog of war," forcing agents to communicate to gain a global view.
+
+### ⚠️ Trade-offs
+
+- **Credit Assignment** — Hard to reward early strategic moves whose payoff arrives many steps later.
+- **Non-Stationarity** — Other agents act simultaneously, making convergence non-trivial.
+
+---
+
+## 📈 Scalability
+
+The architecture scales via **Hierarchical Multi-Agent Systems (HMAS)**. In a city-wide deployment, this 5×5 grid represents a single *Neighborhood*. Each Neighborhood Dispatcher reports to a higher-level **Zone Commander**, allowing the system to manage thousands of nodes without overwhelming any individual agent's context window.
+
+---
+
+## 🚀 Run It Locally
+
+```bash
+git clone https://huggingface.co/spaces/<your-username>/disaster-grid
+cd disaster-grid
+
+python -m venv .venv
+source .venv/bin/activate    # or .venv\Scripts\Activate.ps1 on Windows
+
+pip install -e .
+
+# Run the local CLI simulator
+python -m src.disaster_grid.utils
+
+# Or launch the operational theatre
+python app.py
+```
+
+**Controls in CLI mode:** `W/A/S/D` to move · `R` to repair · `C` to recharge · `Q` to wait · `EXIT` to terminate.
+
+---
+
+## 🏁 Conclusion
+
+The **AI-Kinetic Disaster Recovery Grid** demonstrates that LLMs can transition from *strategic chatting* to **strategic action**. By implementing multi-agent coordination inside a decaying environment, we are engineering the autonomous emergency responders of the future.
+
+---
 
-# ── 6. TRAIN & GRAPH ──
-print("🚀 Starting High-Speed GRPO Training...")
-try:
-    trainer.train()
-finally:
-    # Plotting logic for your final presentation
-    history = trainer.state.log_history
-    steps = [log["step"] for log in history if "reward" in log]
-    rewards = [log["reward"] for log in history if "reward" in log]
-
-    if steps:
-        plt.figure(figsize=(10, 5))
-        plt.plot(steps, rewards, marker='o', color='#10b981', linewidth=2)
-        plt.title('Agent Learning Curve (GRPO Reward over Time)', color='white')
-        plt.xlabel('Steps', color='gray')
-        plt.ylabel('Average Reward', color='gray')
-        plt.gca().set_facecolor('#1e1e1e')
-        plt.gcf().patch.set_facecolor('#1e1e1e')
-        plt.savefig('learning_curve.png', bbox_inches='tight', dpi=300)
-        plt.show()
-        print("✅ Graph saved as 'learning_curve.png'")
-
-# ── 7. SAFE SAVING (Avoiding Mergekit Errors) ──
-# We save as LoRA adapters to avoid needing mergekit
-model.save_pretrained("disaster_agent_lora")
-tokenizer.save_pretrained("disaster_agent_lora")
-print("✅ Training complete. Brain saved to 'disaster_agent_lora'.")2399
+*Developed by **Sanjith P** and **Varsha D** for the Meta PyTorch × Scaler OpenEnv Hackathon.*