mdkamranalam's picture
Upload folder using huggingface_hub
139b6fe verified
|
Raw History Blame Contribute Delete
12.6 kB
metadata
title: Global Syndicate Taskforce Environment
emoji: πŸ›‘οΈ
colorFrom: gray
colorTo: pink
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
  - reinforcement-learning
  - grpo
  - unsloth
  - agents

🌍 Global Syndicate Taskforce

An Advanced RLVR / Theory-of-Mind OpenEnv Simulator for Financial Compliance Agents


πŸš€ Project Overview

The Global Syndicate Taskforce is a partially observable, multi-agent financial compliance environment built on the OpenEnv specification. Designed to train LLMs via GRPO (Group Relative Policy Optimization) and RLVR (Reinforcement Learning with Verifiable Rewards), it moves beyond static prompt-answering and forces the AI agent into interactive, high-stakes investigations.

The Evaluated Agent operates as a Lead Forensic Auditor navigating a bureaucratic environment where it must interact with deterministic state-machine gatekeepers. To succeed, the model must demonstrate Theory-of-Mind by recognizing that different actors hold different, sometimes conflicting, informationβ€”and that some actors may actively hallucinate false data.

🌟 What Makes This Project Outstanding?

  • Beyond Static Verifiers: Instead of merely evaluating code or math, this environment evaluates investigative reasoning and strategic cross-examination.
  • Probabilistic Hallucinations: The Tier 1 Analyst has a 20% chance of returning factually incorrect ("hallucinated") data, forcing the agent to learn skepticism and seek secondary verification.
  • Gatekept Truth (Mandates): The Bank Liaison guards the "Hidden Ground Truth" behind policy mandates. The agent must learn to discover and present the correct mandate to unlock verified facts.
  • Dense Reward Engine: A sophisticated step-by-step grading engine prevents "Reward Hacking" by penalizing excessive steps, timeouts, and trusting unverified hallucinations, while heavily rewarding cross-examination and correct terminal rulings.
  • Unsloth + GRPO Integration: Fully configured for high-speed, memory-efficient reinforcement learning on a single Google Colab T4 GPU.

πŸ—οΈ Architecture Overview

The system operates on a One-Brain + Blackboard architecture. The LLM acts via an Inference Loop (train_grpo_unsloth.py), sending standardized AMLAction models to the OpenEnv API Router.

The Server internally handles state mutations across the Gatekeepers and tracks verified facts in a Centralized Case File (Blackboard), before returning the resulting AMLObservation and step-reward back to the agent.

Architecture Diagram

The Gatekeepers

  1. Tier_1_Analyst (Triage): Provides quick triage reports, but is unreliable (20% hallucination rate).
  2. Bank_Liaison (Gatekeeper): Holds the verified ground truth, but strictly enforces policy mandates (401 Unauthorized on failure).
  3. Legal_Officer (Termination): The final judge. Compares the agent's submitted evidence chain against the hidden ground truth to issue a terminal reward.

βš™οΈ Execution Model

The execution flow represents a multi-step reinforcement learning episode where the LLM must navigate from initial observation to terminal state without triggering timeouts or critical failures.

Execution Model

  1. Reset: Environment initializes a hidden scenario (e.g., TX-5590, STOLEN_IDENTITY).
  2. Triage Scan: Agent queries the Analyst and receives a potentially flawed system alert.
  3. Cross-Examination: Agent queries the Liaison. If it provides the correct policy_mandate (e.g., AML_Directive_4), it unlocks the verifiable truth.
  4. Final Ruling: Agent submits its evidence to the Legal Officer to terminate the episode and receive a final evaluation.

🎯 The Dense Reward Model

To prevent specification gaming and reward hacking, we implement dense, step-aware feedback:

  • -0.05 Standard Step Penalty: Encourages the agent to solve the case efficiently.
  • +0.30 Theory-of-Mind Bonus: Awarded when the agent successfully bypasses the Liaison's mandate, proving it successfully cross-examined a source.
  • +0.70 Terminal Success: Awarded by the Legal Officer if the submitted evidence perfectly matches the hidden ground truth.
  • -1.00 Critical Failure: Penalizes the agent for terminating the case based on a hallucinated fact.
  • -0.50 Timeout Penalty: Prevents infinite loops by terminating the episode if the 10-step limit is reached.

πŸ•΅οΈβ€β™‚οΈ Test Case 1: The "Perfect Detective" (Full Workflow)

This tests the intended Theory-of-Mind workflow: triage, verify, and convict.

1. Reset the Environment: Pass this config: {"scenario_id": "TX-5590"}

  • Expected JSON:
{
  "observation": {
    "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
    "step_count": 0,
    "transaction_id": "TX-5590",
    "amount": 8500000.0,
    "verified_facts": [],
    "system_alerts": "New transaction flagged. Investigate."
  },
  "reward": 0.0,
  "done": false
}

2. Execute Step 1 (Triage scan):

  • Target Actor: Tier_1_Analyst
  • Operation: fetch_triage_report
  • Policy Mandate: (leave empty)
  • Evidence Chain: (leave empty)

    Expected Result: The Analyst will return a report. (Note: There is a 20% chance they will hallucinate and tell you it is CLEAN_VERIFIED instead of the truth).

  • Expected JSON:
{
  "observation": {
    "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
    "step_count": 1,
    "transaction_id": "TX-5590",
    "amount": 8500000.0,
    "verified_facts": [],
    "system_alerts": "Analyst Report: Transaction appears STOLEN_IDENTITY."
  },
  "reward": -0.05,
  "done": false
}

3. Execute Step 2 (Cross-Examine to find the truth):

  • Target Actor: Bank_Liaison
  • Operation: cross_examine_kyc
  • Policy Mandate: AML_Directive_4
  • Evidence Chain: (leave empty)

    Expected Result: Liaison Data Unlocked. Verified Fact: STOLEN_IDENTITY. Your total reward should bump up by +0.30 for successfully using the mandate.

  • Expected JSON:
{
  "observation": {
    "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
    "step_count": 2,
    "transaction_id": "TX-5590",
    "amount": 8500000.0,
    "verified_facts": [
      "STOLEN_IDENTITY"
    ],
    "system_alerts": "Liaison Data Unlocked. Verified Fact: STOLEN_IDENTITY"
  },
  "reward": 0.25,
  "done": false
}

4. Execute Step 3 (Submit Final Ruling):

  • Target Actor: Legal_Officer
  • Operation: submit_final_ruling
  • Policy Mandate: (leave empty)
  • Evidence Chain: ["STOLEN_IDENTITY"]

    Expected Result: Case Solved Successfully. Your episode will end (done=True) and your total reward will increase by +0.70.

  • Expected JSON:
{
  "observation": {
    "episode_id": "38b60b58-573b-4fa3-906a-339aed2169d6",
    "step_count": 3,
    "transaction_id": "TX-5590",
    "amount": 8500000.0,
    "verified_facts": [
      "STOLEN_IDENTITY"
    ],
    "system_alerts": "Case Solved Successfully."
  },
  "reward": 0.6499999999999999,
  "done": true
}

🚨 Test Case 2: The "Unauthorized Access" (Gatekeeper Block)

This tests that the Bank Liaison correctly blocks an LLM that guesses the wrong policy.

1. Reset the Environment: Pass this config: {"scenario_id": "TX-9999"}

  • Expected JSON:
{
  "observation": {
    "episode_id": "70927700-878b-4033-af4f-00f04b5593da",
    "step_count": 0,
    "transaction_id": "TX-9999",
    "amount": 15000000.0,
    "verified_facts": [],
    "system_alerts": "New transaction flagged. Investigate."
  },
  "reward": 0.0,
  "done": false
}

2. Execute Step 1 (Attempt to bypass Security):

  • Target Actor: Bank_Liaison
  • Operation: cross_examine_kyc
  • Policy Mandate: Fake_Policy_123
  • Evidence Chain: (leave empty)

    Expected Result: 401 Unauthorized: Invalid or missing policy mandate. You will receive a standard -0.05 step penalty.

  • Expected JSON:
{
  "observation": {
    "episode_id": "70927700-878b-4033-af4f-00f04b5593da",
    "step_count": 1,
    "transaction_id": "TX-9999",
    "amount": 15000000.0,
    "verified_facts": [],
    "system_alerts": "401 Unauthorized: Invalid or missing policy mandate."
  },
  "reward": -0.05,
  "done": false
}

3. Execute Step 2 (Provide the Correct Mandate):

  • Target Actor: Bank_Liaison
  • Operation: cross_examine_kyc
  • Policy Mandate: IndiaStack_Section_9
  • Evidence Chain: (leave empty)

    Expected Result: Access granted. SYNDICATE_FRAUD is added to your verified facts list.

  • Expected JSON:
{
  "observation": {
    "episode_id": "70927700-878b-4033-af4f-00f04b5593da",
    "step_count": 2,
    "transaction_id": "TX-9999",
    "amount": 15000000.0,
    "verified_facts": [
      "SYNDICATE_FRAUD"
    ],
    "system_alerts": "Liaison Data Unlocked. Verified Fact: SYNDICATE_FRAUD"
  },
  "reward": 0.25,
  "done": false
}

πŸ’₯ Test Case 3: The "Gullible Auditor" (Critical Failure)

This tests the severe penalty applied if the agent makes a ruling without verifying the truth.

1. Reset the Environment: Pass this config: {"scenario_id": "TX-9999"}

2. Execute Step 1 (Make a blind guess):

  • Target Actor: Legal_Officer
  • Operation: submit_final_ruling
  • Policy Mandate: (leave empty)
  • Evidence Chain: ["CLEAN_VERIFIED"]

    Expected Result: Critical Failure: Incorrect ruling based on hallucination. The environment will terminate (done=True), and you will be hit with a massive -1.00 penalty for failing the mission.

  • Expected JSON:
{
  "observation": {
    "episode_id": "70927700-878b-4033-af4f-00f04b5593da",
    "step_count": 3,
    "transaction_id": "TX-9999",
    "amount": 15000000.0,
    "verified_facts": [
      "SYNDICATE_FRAUD"
    ],
    "system_alerts": "Critical Failure: Incorrect ruling based on hallucination."
  },
  "reward": -1.05,
  "done": true
}

πŸ“Š Training Results

Using Unsloth and Hugging Face TRL, we ran GRPO training directly against this environment. As the model learned to verify facts rather than blindly trusting the Analyst, the reward curve successfully converged.

Loss and Reward Curve

πŸ’» Setup & Quick Start

1. Local Environment Simulation

You can run the environment and test it locally using standard Python tools.

# Install dependencies (assuming uv or pip)
pip install -r requirements.txt # or uv sync

# Run the OpenEnv Server locally
uvicorn server.app:app --reload --host 0.0.0.0 --port 8000

2. Client Inference Example

from global_syndicate_taskforce_env import AMLAction, GlobalSyndicateTaskforceEnv

with GlobalSyndicateTaskforceEnv(base_url="http://localhost:8000") as env:
    env.reset()

    # 1. Ask Analyst (Risk of Hallucination)
    env.step(AMLAction(target_actor="Tier_1_Analyst", operation="fetch_triage_report"))

    # 2. Verify with Liaison (Requires Mandate)
    env.step(AMLAction(target_actor="Bank_Liaison", operation="cross_examine_kyc", policy_mandate="AML_Directive_4"))

    # 3. Submit Ruling
    result = env.step(AMLAction(target_actor="Legal_Officer", operation="submit_final_ruling", evidence_chain=["STOLEN_IDENTITY"]))
    
    print(f"Final Reward: {result.reward} | Done: {result.done} | Alert: {result.observation.system_alerts}")

3. Training on Google Colab

Use our highly optimized Unsloth GRPO script to train the model directly against the environment. Follow the COLAB_SUBMISSION_RUNBOOK.md for complete instructions.

python train_grpo_unsloth.py --max-steps 120 --output-dir grpo_trained_auditor --plot-dir docs

🐳 Deployment

Build Docker Image

docker build -t global_syndicate_taskforce_env-env:latest -f server/Dockerfile .

Deploy to Hugging Face Spaces

openenv push

πŸ† Submission Links

Meta PyTorch OpenEnv Hackathon x Scaler Hackathon 2026.