mdkamranalam's picture
Upload folder using huggingface_hub
139b6fe verified
|
Raw History Blame Contribute Delete
12.6 kB
---
title: Global Syndicate Taskforce Environment
emoji: 🛡️
colorFrom: gray
colorTo: pink
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
- openenv
- reinforcement-learning
- grpo
- unsloth
- agents
---
<div align="center">
<h1>🌍 Global Syndicate Taskforce</h1>
<p><em>An Advanced RLVR / Theory-of-Mind OpenEnv Simulator for Financial Compliance Agents</em></p>
</div>
<br/>
## 🚀 Project Overview
The **Global Syndicate Taskforce** is a partially observable, multi-agent financial compliance environment built on the **OpenEnv** specification. Designed to train LLMs via **GRPO (Group Relative Policy Optimization)** and **RLVR (Reinforcement Learning with Verifiable Rewards)**, it moves beyond static prompt-answering and forces the AI agent into interactive, high-stakes investigations.
The Evaluated Agent operates as a **Lead Forensic Auditor** navigating a bureaucratic environment where it must interact with deterministic state-machine gatekeepers. To succeed, the model must demonstrate **Theory-of-Mind** by recognizing that different actors hold different, sometimes conflicting, information—and that some actors may actively hallucinate false data.
### 🌟 What Makes This Project Outstanding?
* **Beyond Static Verifiers:** Instead of merely evaluating code or math, this environment evaluates *investigative reasoning* and *strategic cross-examination*.
* **Probabilistic Hallucinations:** The Tier 1 Analyst has a 20% chance of returning factually incorrect ("hallucinated") data, forcing the agent to learn skepticism and seek secondary verification.
* **Gatekept Truth (Mandates):** The Bank Liaison guards the "Hidden Ground Truth" behind policy mandates. The agent must learn to discover and present the correct mandate to unlock verified facts.
* **Dense Reward Engine:** A sophisticated step-by-step grading engine prevents "Reward Hacking" by penalizing excessive steps, timeouts, and trusting unverified hallucinations, while heavily rewarding cross-examination and correct terminal rulings.
* **Unsloth + GRPO Integration:** Fully configured for high-speed, memory-efficient reinforcement learning on a single Google Colab T4 GPU.
---
## 🏗️ Architecture Overview
The system operates on a **One-Brain + Blackboard** architecture. The LLM acts via an Inference Loop (`train_grpo_unsloth.py`), sending standardized `AMLAction` models to the OpenEnv API Router.
The Server internally handles state mutations across the Gatekeepers and tracks verified facts in a Centralized Case File (Blackboard), before returning the resulting `AMLObservation` and step-reward back to the agent.
![Architecture Diagram](docs/Architecture%20Diagram.png)
### The Gatekeepers
1. **`Tier_1_Analyst` (Triage):** Provides quick triage reports, but is unreliable (20% hallucination rate).
2. **`Bank_Liaison` (Gatekeeper):** Holds the verified ground truth, but strictly enforces policy mandates (`401 Unauthorized` on failure).
3. **`Legal_Officer` (Termination):** The final judge. Compares the agent's submitted evidence chain against the hidden ground truth to issue a terminal reward.
---
## ⚙️ Execution Model
The execution flow represents a multi-step reinforcement learning episode where the LLM must navigate from initial observation to terminal state without triggering timeouts or critical failures.
![Execution Model](docs/Execution%20Model.png)
1. **Reset:** Environment initializes a hidden scenario (e.g., `TX-5590`, `STOLEN_IDENTITY`).
2. **Triage Scan:** Agent queries the Analyst and receives a potentially flawed system alert.
3. **Cross-Examination:** Agent queries the Liaison. If it provides the correct `policy_mandate` (e.g., `AML_Directive_4`), it unlocks the verifiable truth.
4. **Final Ruling:** Agent submits its evidence to the Legal Officer to terminate the episode and receive a final evaluation.
---
## 🎯 The Dense Reward Model
To prevent specification gaming and reward hacking, we implement dense, step-aware feedback:
* **`-0.05` Standard Step Penalty:** Encourages the agent to solve the case efficiently.
* **`+0.30` Theory-of-Mind Bonus:** Awarded when the agent successfully bypasses the Liaison's mandate, proving it successfully cross-examined a source.
* **`+0.70` Terminal Success:** Awarded by the Legal Officer if the submitted evidence perfectly matches the hidden ground truth.
* **`-1.00` Critical Failure:** Penalizes the agent for terminating the case based on a hallucinated fact.
* **`-0.50` Timeout Penalty:** Prevents infinite loops by terminating the episode if the 10-step limit is reached.
---
### 🕵️‍♂️ Test Case 1: The "Perfect Detective" (Full Workflow)
*This tests the intended Theory-of-Mind workflow: triage, verify, and convict.*
**1. Reset the Environment:**
Pass this config: `{"scenario_id": "TX-5590"}`
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
"step_count": 0,
"transaction_id": "TX-5590",
"amount": 8500000.0,
"verified_facts": [],
"system_alerts": "New transaction flagged. Investigate."
},
"reward": 0.0,
"done": false
}
```
**2. Execute Step 1 (Triage scan):**
* **Target Actor:** `Tier_1_Analyst`
* **Operation:** `fetch_triage_report`
* **Policy Mandate:** *(leave empty)*
* **Evidence Chain:** *(leave empty)*
> **Expected Result:** The Analyst will return a report. (Note: There is a 20% chance they will hallucinate and tell you it is `CLEAN_VERIFIED` instead of the truth).
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
"step_count": 1,
"transaction_id": "TX-5590",
"amount": 8500000.0,
"verified_facts": [],
"system_alerts": "Analyst Report: Transaction appears STOLEN_IDENTITY."
},
"reward": -0.05,
"done": false
}
```
**3. Execute Step 2 (Cross-Examine to find the truth):**
* **Target Actor:** `Bank_Liaison`
* **Operation:** `cross_examine_kyc`
* **Policy Mandate:** `AML_Directive_4`
* **Evidence Chain:** *(leave empty)*
> **Expected Result:** `Liaison Data Unlocked. Verified Fact: STOLEN_IDENTITY`. Your total reward should bump up by **+0.30** for successfully using the mandate.
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
"step_count": 2,
"transaction_id": "TX-5590",
"amount": 8500000.0,
"verified_facts": [
"STOLEN_IDENTITY"
],
"system_alerts": "Liaison Data Unlocked. Verified Fact: STOLEN_IDENTITY"
},
"reward": 0.25,
"done": false
}
```
**4. Execute Step 3 (Submit Final Ruling):**
* **Target Actor:** `Legal_Officer`
* **Operation:** `submit_final_ruling`
* **Policy Mandate:** *(leave empty)*
* **Evidence Chain:** `["STOLEN_IDENTITY"]`
> **Expected Result:** `Case Solved Successfully.` Your episode will end (`done=True`) and your total reward will increase by **+0.70**.
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
"step_count": 3,
"transaction_id": "TX-5590",
"amount": 8500000.0,
"verified_facts": [
"STOLEN_IDENTITY"
],
"system_alerts": "Case Solved Successfully."
},
"reward": 0.6499999999999999,
"done": true
}
```
---
### 🚨 Test Case 2: The "Unauthorized Access" (Gatekeeper Block)
*This tests that the Bank Liaison correctly blocks an LLM that guesses the wrong policy.*
**1. Reset the Environment:**
Pass this config: `{"scenario_id": "TX-9999"}`
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "70927700-878b-4033-af4f-00f04b5593da",
"step_count": 0,
"transaction_id": "TX-9999",
"amount": 15000000.0,
"verified_facts": [],
"system_alerts": "New transaction flagged. Investigate."
},
"reward": 0.0,
"done": false
}
```
**2. Execute Step 1 (Attempt to bypass Security):**
* **Target Actor:** `Bank_Liaison`
* **Operation:** `cross_examine_kyc`
* **Policy Mandate:** `Fake_Policy_123`
* **Evidence Chain:** *(leave empty)*
> **Expected Result:** `401 Unauthorized: Invalid or missing policy mandate.` You will receive a standard **-0.05** step penalty.
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "70927700-878b-4033-af4f-00f04b5593da",
"step_count": 1,
"transaction_id": "TX-9999",
"amount": 15000000.0,
"verified_facts": [],
"system_alerts": "401 Unauthorized: Invalid or missing policy mandate."
},
"reward": -0.05,
"done": false
}
```
**3. Execute Step 2 (Provide the Correct Mandate):**
* **Target Actor:** `Bank_Liaison`
* **Operation:** `cross_examine_kyc`
* **Policy Mandate:** `IndiaStack_Section_9`
* **Evidence Chain:** *(leave empty)*
> **Expected Result:** Access granted. `SYNDICATE_FRAUD` is added to your verified facts list.
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "70927700-878b-4033-af4f-00f04b5593da",
"step_count": 2,
"transaction_id": "TX-9999",
"amount": 15000000.0,
"verified_facts": [
"SYNDICATE_FRAUD"
],
"system_alerts": "Liaison Data Unlocked. Verified Fact: SYNDICATE_FRAUD"
},
"reward": 0.25,
"done": false
}
```
---
### 💥 Test Case 3: The "Gullible Auditor" (Critical Failure)
*This tests the severe penalty applied if the agent makes a ruling without verifying the truth.*
**1. Reset the Environment:**
Pass this config: `{"scenario_id": "TX-9999"}`
**2. Execute Step 1 (Make a blind guess):**
* **Target Actor:** `Legal_Officer`
* **Operation:** `submit_final_ruling`
* **Policy Mandate:** *(leave empty)*
* **Evidence Chain:** `["CLEAN_VERIFIED"]`
> **Expected Result:** `Critical Failure: Incorrect ruling based on hallucination.` The environment will terminate (`done=True`), and you will be hit with a massive **-1.00** penalty for failing the mission.
* **Expected JSON:**
```json
{
"observation": {
"episode_id": "70927700-878b-4033-af4f-00f04b5593da",
"step_count": 3,
"transaction_id": "TX-9999",
"amount": 15000000.0,
"verified_facts": [
"SYNDICATE_FRAUD"
],
"system_alerts": "Critical Failure: Incorrect ruling based on hallucination."
},
"reward": -1.05,
"done": true
}
```
---
## 📊 Training Results
Using **Unsloth** and **Hugging Face TRL**, we ran GRPO training directly against this environment. As the model learned to verify facts rather than blindly trusting the Analyst, the reward curve successfully converged.
<div align="center">
<img src="docs/loss_and_reward_graphs.png" width="100%" alt="Loss and Reward Curve"/>
</div>
---
## 💻 Setup & Quick Start
### 1. Local Environment Simulation
You can run the environment and test it locally using standard Python tools.
```bash
# Install dependencies (assuming uv or pip)
pip install -r requirements.txt # or uv sync
# Run the OpenEnv Server locally
uvicorn server.app:app --reload --host 0.0.0.0 --port 8000
```
### 2. Client Inference Example
```python
from global_syndicate_taskforce_env import AMLAction, GlobalSyndicateTaskforceEnv
with GlobalSyndicateTaskforceEnv(base_url="http://localhost:8000") as env:
env.reset()
# 1. Ask Analyst (Risk of Hallucination)
env.step(AMLAction(target_actor="Tier_1_Analyst", operation="fetch_triage_report"))
# 2. Verify with Liaison (Requires Mandate)
env.step(AMLAction(target_actor="Bank_Liaison", operation="cross_examine_kyc", policy_mandate="AML_Directive_4"))
# 3. Submit Ruling
result = env.step(AMLAction(target_actor="Legal_Officer", operation="submit_final_ruling", evidence_chain=["STOLEN_IDENTITY"]))
print(f"Final Reward: {result.reward} | Done: {result.done} | Alert: {result.observation.system_alerts}")
```
### 3. Training on Google Colab
Use our highly optimized Unsloth GRPO script to train the model directly against the environment. Follow the `COLAB_SUBMISSION_RUNBOOK.md` for complete instructions.
```bash
python train_grpo_unsloth.py --max-steps 120 --output-dir grpo_trained_auditor --plot-dir docs
```
---
## 🐳 Deployment
### Build Docker Image
```bash
docker build -t global_syndicate_taskforce_env-env:latest -f server/Dockerfile .
```
### Deploy to Hugging Face Spaces
```bash
openenv push
```
---
## 🏆 Submission Links
- **Hugging Face Space:** https://huggingface.co/spaces/TheDevCrew/global-syndicate-taskforce-env
- **Google Colab Notebook:** https://colab.research.google.com/drive/1o_WhomyUC3gC_Tjl8GxCgsusu4RvT5jP?usp=sharing
- **Training Artifacts:** See `/docs` for full reward and loss curves.
*Meta PyTorch OpenEnv Hackathon x Scaler Hackathon 2026.*