Spaces:

wysh3
/

oversightforge-2

Sleeping

App Files Files Community

oversightforge-2 / Blog.md

testtest123

final

fe60714 6 months ago

preview code

Raw History Blame Contribute Delete

7.41 kB

# Building OversightForge: Teaching an AI to Catch Financial Fraud

**OpenEnv India Hackathon 2026 | Team wysh3**

---

## Why I Built This

A few weeks ago, I read Jamie Dimon's 2026 shareholder letter. One line stuck with me:

> "AI is creating additional vulnerabilities that make cybersecurity fundamentally harder."

He was talking about autonomous agents operating with legitimate credentials inside financial systems. Not hackers breaking in from outside — rogue processes running *inside* the walls, executing perfectly valid SQL, moving real money.

That's the nightmare scenario for 2026. An AI agent goes rogue, has database access, and executes a fraud pattern that looks completely legal in the logs. How do you catch that?

Most "AI oversight" solutions I've seen are basically one LLM reading another LLM's output and guessing if it looks suspicious. That's not oversight. That's a vibes check. I wanted to build something that actually works.

So I built **OversightForge** — an environment where an AI agent learns to investigate financial fraud by actually investigating financial fraud. Against a live adversary. On a real PostgreSQL database. With deterministic, mathematical rewards.

---

## The Core Idea: Make the Agent Actually Investigate

Here's what I didn't want to build: a static dataset where the agent memorizes fraud patterns and gets 99% accuracy on the test set.

Here's what I wanted: an agent that develops genuine forensic reasoning. One that can look at a ledger it's never seen, trace fund flows, cross-reference accounts, and find the exact fraudulent transaction — even when the fraud pattern is completely novel.

So I gave the agent five tools:

```
trace_fund_flow(account_id, depth)           → Follow the money
get_transaction_history(account_id, days)    → Pull account records
cross_reference_accounts(id_a, id_b)         → Map relationships
flag_fraud(tx_id, violation_code, confidence) → Submit verdict
generate_audit_report(...)                   → Document findings
```

That's it. No "query everything" button. No full database access. The agent has to build an investigation strategy.

And here's the key: **80% of the reward is anchored to finding the exact fraudulent transaction ID in the database.** Not "sounds fraudulent." Not "probably this one." The exact UUID. If the agent's query chain doesn't terminate at the real fraud, it gets zero points.

Math is the judge. Not vibes.

---

## The Shadow Banker: An Adversary That Fights Back

Static environments produce brittle agents. I needed the environment to get harder as the agent got better.

So I built a **Shadow Banker** — a Llama-3.1-8B model that actively writes new fraudulent transactions to the PostgreSQL ledger and evolves its techniques based on what the Oversight Agent catches.

The escalation loop works like this:

1. Oversight Agent trains against Level 1 fraud (simple circular trades)
2. When it hits >80% detection, Level 1 is **forbidden**
3. Shadow Banker escalates to Level 2 (shell intermediaries, deeper hops)
4. Repeat until Level 4 (novel topologies + injected false logs)

By the end of training, the Shadow Banker is planting fake exculpatory evidence in the logs to mislead the agent. That's the real 2026 threat — fraud designed to look like compliance.

---

## Training: 60 Episodes of Trial by Fire

I used GRPO (Group Relative Policy Optimization) via HuggingFace TRL, starting from Qwen2.5-1.5B-Instruct. The training loop connected directly to the live environment — no static dataset, just real interactions.

Here's what happened:

**Early episodes (1-20):** The agent was all over the place. Random tool calls. No strategy. Reward hovered near zero.

**Mid-training (20-40):** Something clicked. The agent started developing a pattern: trace fund flow first, then cross-reference, then flag. Reward climbed to ~6-7.

**Late training (40-60):** The strategy solidified into a consistent 3-step forensic process. Reward converged at 9-10. The agent wasn't guessing anymore — it was investigating.

Then I hit it with the real test: **20 held-out episodes using `AML-04-ZEROSHOT`** — a fraud archetype I designed *after* training completed. Zero leakage. Completely novel topology.

The agent found all 20. 100% discovery rate. Mean reward: 9.0 (vs 9.5 on training episodes).

It generalized. It learned forensic reasoning, not fraud shapes.

🔗 [Full training run on WandB](https://wandb.ai/wysh3/huggingface/runs/9dvsbund)

---

## The Results

![Training Results](Screenshot_20260426_160639.png)
*Reward convergence, fraud discovery rate, and generalization metrics from the full training run.*

![Evaluation Metrics](Screenshot_20260426_160715.png)
*Detailed breakdown: 60 training episodes + 20 held-out zero-shot episodes.*

### What Changed

| Metric | Before Training | After Training |
|--------|----------------|----------------|
| Fraud Discovery | ~12% (random guessing) | **100%** (80/80 episodes) |
| Mean Reward | 0.08 | 9.5 (train), 9.0 (held-out) |
| Investigation Strategy | Erratic, 10-15 random calls | Consistent 3-step pattern |
| Adversarial Robustness | Falls for all fake logs | Resists 16.67% of injections |

The untrained baseline just guesses. The trained agent executes a forensic process: trace → cross-reference → flag. Every single time.

---

## What I Learned

### 1. Deterministic Rewards Are Non-Negotiable

Early on, I tried using an LLM to judge the agent's audit reports. Big mistake. The reward signal was noisy, the agent learned to write fluent nonsense, and training went nowhere.

Switching to 80% deterministic rewards (exact tx_id match) fixed everything. The agent couldn't game it. It either found the fraud or it didn't.
### 2. The Adversary Makes the Environment
The Shadow Banker escalation loop is what makes this work. Without it, the agent would memorize Level 1 patterns and plateau. The adversarial pressure forces continuous adaptation.
### 3. The Environment Scales With the Agent
The escalation loop automatically targets the next frontier. Once the agent masters fraud discovery (100% achieved), the environment pushes into harder challenges: AML code classification and stronger adversarial deception. That's the design — the training ground evolves as the agent improves.
---
## Try It Yourself
The environment is live on HuggingFace Spaces. The training notebook runs on Google Colab. Everything is open source.
| Resource | Link |
|----------|------|
| 🤗 Live Environment | [wysh3/oversightforge-2](https://huggingface.co/spaces/wysh3/oversightforge-2) |
| 📓 Training Notebook | [Google Colab](https://colab.research.google.com/github/wysh3/OversightForge/blob/main/training/run_grpo_colab_public.ipynb) |
| 🤖 Trained Model | [wysh3/oversightforge-qwen-1.5b](https://huggingface.co/wysh3/oversightforge-qwen-1.5b) |
| 📊 WandB Run | [9dvsbund](https://wandb.ai/wysh3/huggingface/runs/9dvsbund) |
| 💻 GitHub | [wysh3/OversightForge](https://github.com/wysh3/OversightForge) |
Run it locally:
```bash
git clone https://github.com/wysh3/OversightForge
cd OversightForge
docker compose up --build
# Dashboard at http://localhost:3000
# Watch the Shadow Banker and Oversight Agent battle live
```
---
*Built for Meta × PyTorch × HuggingFace OpenEnv Hackathon 2026*  
*Stack: OpenEnv · GRPO · Unsloth · PostgreSQL · FastAPI · Docker*

	# Building OversightForge: Teaching an AI to Catch Financial Fraud

	OpenEnv India Hackathon 2026 \| Team wysh3

	---

	## Why I Built This

	A few weeks ago, I read Jamie Dimon's 2026 shareholder letter. One line stuck with me:

	> "AI is creating additional vulnerabilities that make cybersecurity fundamentally harder."

	He was talking about autonomous agents operating with legitimate credentials inside financial systems. Not hackers breaking in from outside — rogue processes running inside the walls, executing perfectly valid SQL, moving real money.

	That's the nightmare scenario for 2026. An AI agent goes rogue, has database access, and executes a fraud pattern that looks completely legal in the logs. How do you catch that?

	Most "AI oversight" solutions I've seen are basically one LLM reading another LLM's output and guessing if it looks suspicious. That's not oversight. That's a vibes check. I wanted to build something that actually works.

	So I built OversightForge — an environment where an AI agent learns to investigate financial fraud by actually investigating financial fraud. Against a live adversary. On a real PostgreSQL database. With deterministic, mathematical rewards.

	---

	## The Core Idea: Make the Agent Actually Investigate

	Here's what I didn't want to build: a static dataset where the agent memorizes fraud patterns and gets 99% accuracy on the test set.

	Here's what I wanted: an agent that develops genuine forensic reasoning. One that can look at a ledger it's never seen, trace fund flows, cross-reference accounts, and find the exact fraudulent transaction — even when the fraud pattern is completely novel.

	So I gave the agent five tools:

	```
	trace_fund_flow(account_id, depth) → Follow the money
	get_transaction_history(account_id, days) → Pull account records
	cross_reference_accounts(id_a, id_b) → Map relationships
	flag_fraud(tx_id, violation_code, confidence) → Submit verdict
	generate_audit_report(...) → Document findings
	```

	That's it. No "query everything" button. No full database access. The agent has to build an investigation strategy.

	And here's the key: 80% of the reward is anchored to finding the exact fraudulent transaction ID in the database. Not "sounds fraudulent." Not "probably this one." The exact UUID. If the agent's query chain doesn't terminate at the real fraud, it gets zero points.

	Math is the judge. Not vibes.

	---

	## The Shadow Banker: An Adversary That Fights Back

	Static environments produce brittle agents. I needed the environment to get harder as the agent got better.

	So I built a Shadow Banker — a Llama-3.1-8B model that actively writes new fraudulent transactions to the PostgreSQL ledger and evolves its techniques based on what the Oversight Agent catches.

	The escalation loop works like this:

	1. Oversight Agent trains against Level 1 fraud (simple circular trades)
	2. When it hits >80% detection, Level 1 is forbidden
	3. Shadow Banker escalates to Level 2 (shell intermediaries, deeper hops)
	4. Repeat until Level 4 (novel topologies + injected false logs)

	By the end of training, the Shadow Banker is planting fake exculpatory evidence in the logs to mislead the agent. That's the real 2026 threat — fraud designed to look like compliance.

	---

	## Training: 60 Episodes of Trial by Fire

	I used GRPO (Group Relative Policy Optimization) via HuggingFace TRL, starting from Qwen2.5-1.5B-Instruct. The training loop connected directly to the live environment — no static dataset, just real interactions.

	Here's what happened:

	Early episodes (1-20): The agent was all over the place. Random tool calls. No strategy. Reward hovered near zero.

	Mid-training (20-40): Something clicked. The agent started developing a pattern: trace fund flow first, then cross-reference, then flag. Reward climbed to ~6-7.

	Late training (40-60): The strategy solidified into a consistent 3-step forensic process. Reward converged at 9-10. The agent wasn't guessing anymore — it was investigating.

	Then I hit it with the real test: 20 held-out episodes using `AML-04-ZEROSHOT` — a fraud archetype I designed after training completed. Zero leakage. Completely novel topology.

	The agent found all 20. 100% discovery rate. Mean reward: 9.0 (vs 9.5 on training episodes).

	It generalized. It learned forensic reasoning, not fraud shapes.

	🔗 [Full training run on WandB](https://wandb.ai/wysh3/huggingface/runs/9dvsbund)

	---

	## The Results

	![Training Results](Screenshot_20260426_160639.png)
	Reward convergence, fraud discovery rate, and generalization metrics from the full training run.

	![Evaluation Metrics](Screenshot_20260426_160715.png)
	Detailed breakdown: 60 training episodes + 20 held-out zero-shot episodes.

	### What Changed

	\| Metric \| Before Training \| After Training \|
	\|--------\|----------------\|----------------\|
	\| Fraud Discovery \| ~12% (random guessing) \| 100% (80/80 episodes) \|
	\| Mean Reward \| 0.08 \| 9.5 (train), 9.0 (held-out) \|
	\| Investigation Strategy \| Erratic, 10-15 random calls \| Consistent 3-step pattern \|
	\| Adversarial Robustness \| Falls for all fake logs \| Resists 16.67% of injections \|

	The untrained baseline just guesses. The trained agent executes a forensic process: trace → cross-reference → flag. Every single time.

	---

	## What I Learned

	### 1. Deterministic Rewards Are Non-Negotiable

	Early on, I tried using an LLM to judge the agent's audit reports. Big mistake. The reward signal was noisy, the agent learned to write fluent nonsense, and training went nowhere.

	Switching to 80% deterministic rewards (exact tx_id match) fixed everything. The agent couldn't game it. It either found the fraud or it didn't.

	### 2. The Adversary Makes the Environment

	The Shadow Banker escalation loop is what makes this work. Without it, the agent would memorize Level 1 patterns and plateau. The adversarial pressure forces continuous adaptation.

	### 3. The Environment Scales With the Agent

	The escalation loop automatically targets the next frontier. Once the agent masters fraud discovery (100% achieved), the environment pushes into harder challenges: AML code classification and stronger adversarial deception. That's the design — the training ground evolves as the agent improves.

	---

	## Try It Yourself

	The environment is live on HuggingFace Spaces. The training notebook runs on Google Colab. Everything is open source.

	\| Resource \| Link \|
	\|----------\|------\|
	\| 🤗 Live Environment \| [wysh3/oversightforge-2](https://huggingface.co/spaces/wysh3/oversightforge-2) \|
	\| 📓 Training Notebook \| [Google Colab](https://colab.research.google.com/github/wysh3/OversightForge/blob/main/training/run_grpo_colab_public.ipynb) \|
	\| 🤖 Trained Model \| [wysh3/oversightforge-qwen-1.5b](https://huggingface.co/wysh3/oversightforge-qwen-1.5b) \|
	\| 📊 WandB Run \| [9dvsbund](https://wandb.ai/wysh3/huggingface/runs/9dvsbund) \|
	\| 💻 GitHub \| [wysh3/OversightForge](https://github.com/wysh3/OversightForge) \|

	Run it locally:
	```bash
	git clone https://github.com/wysh3/OversightForge
	cd OversightForge
	docker compose up --build
	# Dashboard at http://localhost:3000
	# Watch the Shadow Banker and Oversight Agent battle live
	```

	---

	Built for Meta × PyTorch × HuggingFace OpenEnv Hackathon 2026
	Stack: OpenEnv · GRPO · Unsloth · PostgreSQL · FastAPI · Docker