--- title: Global Syndicate Taskforce Environment emoji: πŸ›‘οΈ colorFrom: gray colorTo: pink sdk: docker pinned: false app_port: 8000 base_path: /web tags: - openenv - reinforcement-learning - grpo - unsloth - agents ---

🌍 Global Syndicate Taskforce

An Advanced RLVR / Theory-of-Mind OpenEnv Simulator for Financial Compliance Agents


## πŸš€ Project Overview The **Global Syndicate Taskforce** is a partially observable, multi-agent financial compliance environment built on the **OpenEnv** specification. Designed to train LLMs via **GRPO (Group Relative Policy Optimization)** and **RLVR (Reinforcement Learning with Verifiable Rewards)**, it moves beyond static prompt-answering and forces the AI agent into interactive, high-stakes investigations. The Evaluated Agent operates as a **Lead Forensic Auditor** navigating a bureaucratic environment where it must interact with deterministic state-machine gatekeepers. To succeed, the model must demonstrate **Theory-of-Mind** by recognizing that different actors hold different, sometimes conflicting, informationβ€”and that some actors may actively hallucinate false data. ### 🌟 What Makes This Project Outstanding? * **Beyond Static Verifiers:** Instead of merely evaluating code or math, this environment evaluates *investigative reasoning* and *strategic cross-examination*. * **Probabilistic Hallucinations:** The Tier 1 Analyst has a 20% chance of returning factually incorrect ("hallucinated") data, forcing the agent to learn skepticism and seek secondary verification. * **Gatekept Truth (Mandates):** The Bank Liaison guards the "Hidden Ground Truth" behind policy mandates. The agent must learn to discover and present the correct mandate to unlock verified facts. * **Dense Reward Engine:** A sophisticated step-by-step grading engine prevents "Reward Hacking" by penalizing excessive steps, timeouts, and trusting unverified hallucinations, while heavily rewarding cross-examination and correct terminal rulings. * **Unsloth + GRPO Integration:** Fully configured for high-speed, memory-efficient reinforcement learning on a single Google Colab T4 GPU. --- ## πŸ—οΈ Architecture Overview The system operates on a **One-Brain + Blackboard** architecture. The LLM acts via an Inference Loop (`train_grpo_unsloth.py`), sending standardized `AMLAction` models to the OpenEnv API Router. The Server internally handles state mutations across the Gatekeepers and tracks verified facts in a Centralized Case File (Blackboard), before returning the resulting `AMLObservation` and step-reward back to the agent. ![Architecture Diagram](docs/Architecture%20Diagram.png) ### The Gatekeepers 1. **`Tier_1_Analyst` (Triage):** Provides quick triage reports, but is unreliable (20% hallucination rate). 2. **`Bank_Liaison` (Gatekeeper):** Holds the verified ground truth, but strictly enforces policy mandates (`401 Unauthorized` on failure). 3. **`Legal_Officer` (Termination):** The final judge. Compares the agent's submitted evidence chain against the hidden ground truth to issue a terminal reward. --- ## βš™οΈ Execution Model The execution flow represents a multi-step reinforcement learning episode where the LLM must navigate from initial observation to terminal state without triggering timeouts or critical failures. ![Execution Model](docs/Execution%20Model.png) 1. **Reset:** Environment initializes a hidden scenario (e.g., `TX-5590`, `STOLEN_IDENTITY`). 2. **Triage Scan:** Agent queries the Analyst and receives a potentially flawed system alert. 3. **Cross-Examination:** Agent queries the Liaison. If it provides the correct `policy_mandate` (e.g., `AML_Directive_4`), it unlocks the verifiable truth. 4. **Final Ruling:** Agent submits its evidence to the Legal Officer to terminate the episode and receive a final evaluation. --- ## 🎯 The Dense Reward Model To prevent specification gaming and reward hacking, we implement dense, step-aware feedback: * **`-0.05` Standard Step Penalty:** Encourages the agent to solve the case efficiently. * **`+0.30` Theory-of-Mind Bonus:** Awarded when the agent successfully bypasses the Liaison's mandate, proving it successfully cross-examined a source. * **`+0.70` Terminal Success:** Awarded by the Legal Officer if the submitted evidence perfectly matches the hidden ground truth. * **`-1.00` Critical Failure:** Penalizes the agent for terminating the case based on a hallucinated fact. * **`-0.50` Timeout Penalty:** Prevents infinite loops by terminating the episode if the 10-step limit is reached. --- ### πŸ•΅οΈβ€β™‚οΈ Test Case 1: The "Perfect Detective" (Full Workflow) *This tests the intended Theory-of-Mind workflow: triage, verify, and convict.* **1. Reset the Environment:** Pass this config: `{"scenario_id": "TX-5590"}` * **Expected JSON:** ```json { "observation": { "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6", "step_count": 0, "transaction_id": "TX-5590", "amount": 8500000.0, "verified_facts": [], "system_alerts": "New transaction flagged. Investigate." }, "reward": 0.0, "done": false } ``` **2. Execute Step 1 (Triage scan):** * **Target Actor:** `Tier_1_Analyst` * **Operation:** `fetch_triage_report` * **Policy Mandate:** *(leave empty)* * **Evidence Chain:** *(leave empty)* > **Expected Result:** The Analyst will return a report. (Note: There is a 20% chance they will hallucinate and tell you it is `CLEAN_VERIFIED` instead of the truth). * **Expected JSON:** ```json { "observation": { "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6", "step_count": 1, "transaction_id": "TX-5590", "amount": 8500000.0, "verified_facts": [], "system_alerts": "Analyst Report: Transaction appears STOLEN_IDENTITY." }, "reward": -0.05, "done": false } ``` **3. Execute Step 2 (Cross-Examine to find the truth):** * **Target Actor:** `Bank_Liaison` * **Operation:** `cross_examine_kyc` * **Policy Mandate:** `AML_Directive_4` * **Evidence Chain:** *(leave empty)* > **Expected Result:** `Liaison Data Unlocked. Verified Fact: STOLEN_IDENTITY`. Your total reward should bump up by **+0.30** for successfully using the mandate. * **Expected JSON:** ```json { "observation": { "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6", "step_count": 2, "transaction_id": "TX-5590", "amount": 8500000.0, "verified_facts": [ "STOLEN_IDENTITY" ], "system_alerts": "Liaison Data Unlocked. Verified Fact: STOLEN_IDENTITY" }, "reward": 0.25, "done": false } ``` **4. Execute Step 3 (Submit Final Ruling):** * **Target Actor:** `Legal_Officer` * **Operation:** `submit_final_ruling` * **Policy Mandate:** *(leave empty)* * **Evidence Chain:** `["STOLEN_IDENTITY"]` > **Expected Result:** `Case Solved Successfully.` Your episode will end (`done=True`) and your total reward will increase by **+0.70**. * **Expected JSON:** ```json { "observation": { "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6", "step_count": 3, "transaction_id": "TX-5590", "amount": 8500000.0, "verified_facts": [ "STOLEN_IDENTITY" ], "system_alerts": "Case Solved Successfully." }, "reward": 0.6499999999999999, "done": true } ``` --- ### 🚨 Test Case 2: The "Unauthorized Access" (Gatekeeper Block) *This tests that the Bank Liaison correctly blocks an LLM that guesses the wrong policy.* **1. Reset the Environment:** Pass this config: `{"scenario_id": "TX-9999"}` * **Expected JSON:** ```json { "observation": { "episode_id": "70927700-878b-4033-af4f-00f04b5593da", "step_count": 0, "transaction_id": "TX-9999", "amount": 15000000.0, "verified_facts": [], "system_alerts": "New transaction flagged. Investigate." }, "reward": 0.0, "done": false } ``` **2. Execute Step 1 (Attempt to bypass Security):** * **Target Actor:** `Bank_Liaison` * **Operation:** `cross_examine_kyc` * **Policy Mandate:** `Fake_Policy_123` * **Evidence Chain:** *(leave empty)* > **Expected Result:** `401 Unauthorized: Invalid or missing policy mandate.` You will receive a standard **-0.05** step penalty. * **Expected JSON:** ```json { "observation": { "episode_id": "70927700-878b-4033-af4f-00f04b5593da", "step_count": 1, "transaction_id": "TX-9999", "amount": 15000000.0, "verified_facts": [], "system_alerts": "401 Unauthorized: Invalid or missing policy mandate." }, "reward": -0.05, "done": false } ``` **3. Execute Step 2 (Provide the Correct Mandate):** * **Target Actor:** `Bank_Liaison` * **Operation:** `cross_examine_kyc` * **Policy Mandate:** `IndiaStack_Section_9` * **Evidence Chain:** *(leave empty)* > **Expected Result:** Access granted. `SYNDICATE_FRAUD` is added to your verified facts list. * **Expected JSON:** ```json { "observation": { "episode_id": "70927700-878b-4033-af4f-00f04b5593da", "step_count": 2, "transaction_id": "TX-9999", "amount": 15000000.0, "verified_facts": [ "SYNDICATE_FRAUD" ], "system_alerts": "Liaison Data Unlocked. Verified Fact: SYNDICATE_FRAUD" }, "reward": 0.25, "done": false } ``` --- ### πŸ’₯ Test Case 3: The "Gullible Auditor" (Critical Failure) *This tests the severe penalty applied if the agent makes a ruling without verifying the truth.* **1. Reset the Environment:** Pass this config: `{"scenario_id": "TX-9999"}` **2. Execute Step 1 (Make a blind guess):** * **Target Actor:** `Legal_Officer` * **Operation:** `submit_final_ruling` * **Policy Mandate:** *(leave empty)* * **Evidence Chain:** `["CLEAN_VERIFIED"]` > **Expected Result:** `Critical Failure: Incorrect ruling based on hallucination.` The environment will terminate (`done=True`), and you will be hit with a massive **-1.00** penalty for failing the mission. * **Expected JSON:** ```json { "observation": { "episode_id": "70927700-878b-4033-af4f-00f04b5593da", "step_count": 3, "transaction_id": "TX-9999", "amount": 15000000.0, "verified_facts": [ "SYNDICATE_FRAUD" ], "system_alerts": "Critical Failure: Incorrect ruling based on hallucination." }, "reward": -1.05, "done": true } ``` --- ## πŸ“Š Training Results Using **Unsloth** and **Hugging Face TRL**, we ran GRPO training directly against this environment. As the model learned to verify facts rather than blindly trusting the Analyst, the reward curve successfully converged.
Loss and Reward Curve
--- ## πŸ’» Setup & Quick Start ### 1. Local Environment Simulation You can run the environment and test it locally using standard Python tools. ```bash # Install dependencies (assuming uv or pip) pip install -r requirements.txt # or uv sync # Run the OpenEnv Server locally uvicorn server.app:app --reload --host 0.0.0.0 --port 8000 ``` ### 2. Client Inference Example ```python from global_syndicate_taskforce_env import AMLAction, GlobalSyndicateTaskforceEnv with GlobalSyndicateTaskforceEnv(base_url="http://localhost:8000") as env: env.reset() # 1. Ask Analyst (Risk of Hallucination) env.step(AMLAction(target_actor="Tier_1_Analyst", operation="fetch_triage_report")) # 2. Verify with Liaison (Requires Mandate) env.step(AMLAction(target_actor="Bank_Liaison", operation="cross_examine_kyc", policy_mandate="AML_Directive_4")) # 3. Submit Ruling result = env.step(AMLAction(target_actor="Legal_Officer", operation="submit_final_ruling", evidence_chain=["STOLEN_IDENTITY"])) print(f"Final Reward: {result.reward} | Done: {result.done} | Alert: {result.observation.system_alerts}") ``` ### 3. Training on Google Colab Use our highly optimized Unsloth GRPO script to train the model directly against the environment. Follow the `COLAB_SUBMISSION_RUNBOOK.md` for complete instructions. ```bash python train_grpo_unsloth.py --max-steps 120 --output-dir grpo_trained_auditor --plot-dir docs ``` --- ## 🐳 Deployment ### Build Docker Image ```bash docker build -t global_syndicate_taskforce_env-env:latest -f server/Dockerfile . ``` ### Deploy to Hugging Face Spaces ```bash openenv push ``` --- ## πŸ† Submission Links - **Hugging Face Space:** https://huggingface.co/spaces/TheDevCrew/global-syndicate-taskforce-env - **Google Colab Notebook:** https://colab.research.google.com/drive/1o_WhomyUC3gC_Tjl8GxCgsusu4RvT5jP?usp=sharing - **Training Artifacts:** See `/docs` for full reward and loss curves. *Meta PyTorch OpenEnv Hackathon x Scaler Hackathon 2026.*