Remove BLOG_DRAFT.md and ROUND2_PROBLEM_STATEMENT.md

BLOG_DRAFT.md

@@ -1,216 +0,0 @@
-# Invoice Processing Pipeline — Multi-Agent RL Environment for Financial Fraud Detection
-
-**Meta PyTorch OpenEnv Hackathon Grand Finale | April 25–26, 2026**
-**Team: Pritam Satpathy + Gnana Nawin T**
-
----
-
-## The Problem
-
-Invoice fraud costs businesses an estimated **5% of annual revenue**. Finance teams manually process thousands of invoices every month — extracting vendor names, dates, line items, totals — and checking them against purchase orders for discrepancies. This is:
-
-- Slow (hours per batch)
-- Error-prone (typos, OCR noise, format chaos)
-- Gameable (phantom vendors, price gouging, duplicate submissions)
-
-We built an **RL training environment** that teaches LLMs to do this automatically — and improves itself when it finds its own blind spots.
-
----
-
-## What We Built
-
-An OpenEnv-compatible environment deployed on HuggingFace Spaces:
-**https://ps2181-invoice-processing-pipeline.hf.space**
-
-### 5-Agent Architecture
-
-```
-Generator ──► Extractor ──► Auditor ──► Approver
-                                ▲
-                           Regulator
-                        (cross-episode)
-```
-
-| Agent | Role | Reward Signal |
-|-------|------|--------------|
-| **Generator** | Creates clean or fraudulent invoices | Rewarded when fraud slips past Auditor (adversarial self-play) |
-| **Extractor** | Reads raw invoice text → structured JSON | 4 independent signals: format, field accuracy, math consistency, completeness |
-| **Auditor** | Reviews extraction, flags fraud | +0.99 correct detection, +0.90 clean clearance, 0.01 for miss or false positive |
-| **Approver** | Final approve/reject/escalate decision | +0.95 correct decision |
-| **Regulator** | Monitors Auditor blind spots **across episodes** | Precision + recall of blind spot predictions |
-
-### The Key Innovation: The Regulator
-
-The Regulator is a **cross-episode meta-agent** — it watches the Auditor's decision history over 30 episodes and identifies systematic failure patterns:
-
-```
-AUDITOR PERFORMANCE TRACKER (last 30 episodes)
-
-Fraud Type            Detection Rate
-─────────────────────────────────────
-phantom_vendor        31%   ⚠ BLIND SPOT
-price_gouging         74%   ✓ OK
-math_fraud            81%   ✓ OK
-duplicate_submission  62%   ✓ OK
-
-False Positive Rate:  12%   ✓ OK
-
-REGULATOR VERDICT: Recommend retraining on phantom_vendor
-```
-
-When the Regulator detects a blind spot, the **Generator automatically starts producing more of that fraud type** — closing the self-improvement loop without human intervention.
-
-This directly addresses **Theme #1 (Fleet AI Scalable Oversight)** and **Theme #4 (Self-Improvement)**.
-
----
-
-## 7 Tasks (Progressive Difficulty)
-
-| Task | Difficulty | What the Agent Does |
-|------|-----------|---------------------|
-| `easy` | Easy | Extract fields from a single clean invoice |
-| `medium` | Medium | Clean + normalise a batch of messy invoices (typos, date chaos, currency symbols) |
-| `hard` | Hard | Extract + reconcile against purchase orders, flag discrepancies |
-| `expert` | Expert | Fraud audit: classify phantom_vendor / price_gouging / math_fraud / duplicate_submission |
-| `adversarial` | Hard | Extract from OCR-corrupted invoice with SUBTOTAL trap and FX noise lines |
-| `negotiate` | Medium | Ask clarification questions then submit extraction (bonus for ≤2 questions) |
-| `supply_chain` | Expert | Detect quantity shortfalls, price spikes, phantom deliveries in delivery records |
-
----
-
-## Design Decisions
-
-### 4 Independent Reward Functions (Anti-Hacking)
-
-Per the hackathon guide Section 7: *"use multiple independent reward functions — if you only have one reward signal, it is easier for the model to hack it."*
-
-```python
-format_reward()       # Are all 5 required JSON keys present?       weight: 0.10
-field_reward()        # Do vendor/date/currency/total match?         weight: 0.40
-math_reward()         # Does qty × unit_price = amount for all items? weight: 0.25
-completeness_reward() # Are all line items present (recall)?          weight: 0.25
-```
-
-During training we observed the model maximising `math_reward` (0.97) and `completeness_reward` (1.0) while `field_reward` stayed at 0.0 — the model learned to output arithmetic-consistent JSON while hallucinating values. **Our independent signals made this reward hacking immediately visible**, confirming the design choice.
-
-### Adversarial Self-Play
-
-The Generator is rewarded when its fraud evades the Auditor:
-- Fraud fully undetected by Auditor + Approver approves → Generator reward: **0.85**
-- Auditor missed but Approver caught → Generator reward: **0.60**
-- Auditor caught it → Generator reward: **0.10**
-
-This creates evolutionary pressure: the Generator evolves harder-to-detect fraud patterns, forcing the Auditor to improve.
-
-### Dynamic Difficulty
-
-The environment tracks recent agent scores per task (rolling window of 10 episodes) and adjusts generation parameters:
-- Agent scoring ≥ 0.85 → harder parameters (more invoices, more OCR noise, more discrepancies)
-- Agent scoring < 0.60 → easier parameters
-- In between → standard
-
-### All Rewards Clamped to (0.01, 0.99)
-
-Rewards are never exactly 0 or 1 — avoids log(0) in policy gradient and prevents the model from getting stuck at boundaries.
-
----
-
-## Tech Stack
-
-```
-Environment:  FastAPI + OpenEnv-core + Pydantic
-Deployment:   HuggingFace Spaces (Docker, port 7860)
-UI:           Gradio (mounted at /web)
-Training:     TRL GRPOTrainer + Unsloth (Qwen2.5-1.5B-Instruct, 4-bit QLoRA)
-Model:        unsloth/Qwen2.5-1.5B-Instruct  r=16 LoRA
-Reward:       4 local signals + live /grader endpoint on HF Space
-```
-
----
-
-## Training Setup
-
-**GRPO (Group Relative Policy Optimization)** with:
-- `num_generations = 4` — 4 completions per prompt, compared within group
-- `max_steps = 200`
-- `learning_rate = 5e-6`
-- Live `/grader` endpoint on HF Space as environment verifier
-
-The training loop:
-```
-Colab samples episode → HF Space /reset → gets live invoice
-Model generates JSON extraction
-HF Space /grader scores it against ground truth
-GRPO updates model toward higher-scoring completions
-```
-
-### Reward Curve
-
-| Step | Total Reward | Env Score | Format | Math |
-|------|-------------|-----------|--------|------|
-| 10   | 2.361       | 0.113     | 0.900  | 0.347 |
-| 20   | 2.595       | 0.282     | 0.900  | 0.413 |
-| 30   | 2.657       | 0.304     | 0.950  | 0.403 |
-
-Environment score rose **0.113 → 0.304 in 30 steps** — a 169% improvement in the model's ability to correctly extract invoice data as scored by the live environment grader.
-
-*[Add reward curve plot image here]*
-
----
-
-## API Endpoints
-
-| Endpoint | Method | Description |
-|----------|--------|-------------|
-| `/health` | GET | Health check |
-| `/reset` | POST | Start new episode `{"task_id": "easy"}` |
-| `/step` | POST | Submit extraction, get reward + feedback |
-| `/grader` | POST | Score without consuming attempt |
-| `/state` | GET | Episode metadata |
-| `/tasks` | GET | List all 7 tasks with schemas |
-| `/ws` | WebSocket | Full episode over WebSocket (OpenEnv standard) |
-| `/web` | GET | Gradio interactive UI |
-
----
-
-## What Makes This Novel
-
-1. **Regulator agent** — no other OpenEnv environment has a cross-episode meta-agent that monitors another agent for systematic cognitive blind spots
-
-2. **Closed self-improvement loop** — Regulator detects blind spot → Generator biases fraud generation toward that type → Auditor forced to improve → no human intervention required
-
-3. **Adversarial Generator arms race** — Generator rewarded for evading Auditor creates evolutionary pressure on fraud detection
-
-4. **Live environment as verifier** — training Colab directly calls `/grader` on deployed HF Space — the environment IS the reward function
-
-5. **4 independent reward signals** — made reward hacking immediately visible during training (detected it at step 10)
-
----
-
-## Theme Alignment
-
-| Theme | Alignment |
-|-------|-----------|
-| **#1 Multi-Agent** | 5 agents with conflicting incentives |
-| **#1 Sub: Fleet AI Oversight** (bonus) | Regulator monitors Auditor cross-episode |
-| **#3.1 Professional Tasks** | Invoice processing = core enterprise workflow |
-| **#3.1 Sub: Scaler AI Labs** (bonus) | Multi-agent RL for enterprise financial workflows |
-| **#4 Self-Improvement** | Generator adapts based on Regulator blind spot findings |
-
----
-
-## Links
-
-- **Live Environment:** https://ps2181-invoice-processing-pipeline.hf.space
-- **Gradio UI:** https://ps2181-invoice-processing-pipeline.hf.space/web
-- **API Docs:** https://ps2181-invoice-processing-pipeline.hf.space/docs
-- **GitHub:** https://github.com/ps2181/invoice-processing-pipeline
-- **Training Colab:** *[add link after saving to GitHub]*
-
----
-
-## Team
-
-**Pritam Satpathy** + **Gnana Nawin T**
-Meta PyTorch OpenEnv Hackathon Grand Finale
-Scaler School of Technology, Bangalore — April 25–26, 2026

ROUND2_PROBLEM_STATEMENT.md

@@ -1,182 +0,0 @@
-# Round 2 Problem Statement
-
-## Title
-
-Autonomous Accounts Payable Control Tower
-
-## Theme Alignment
-
-This environment intentionally covers more than one Round 2 theme:
-
-- Multi-Agent Interactions
-- Long-Horizon Planning and Instruction Following
-- World Modeling across professional tasks
-- Self-Improving agent systems
-
-## Core Problem Statement
-
-Design an OpenEnv environment in which one or more AI agents operate as an accounts-payable control tower for a mid-sized company. The agent must transform noisy financial documents into reliable, auditable decisions across invoice extraction, normalization, reconciliation, fraud review, clarification, and downstream supply-chain checks.
-
-The real-world challenge is not simple OCR or single-step parsing. A strong agent must maintain working memory across turns, choose when to ask for clarification, compare multiple sources of truth, detect subtle inconsistencies, and maximize business-safe outcomes under limited attempts.
-
-In practical terms, the environment simulates the day-to-day work of finance operations teams, procurement reviewers, and audit analysts who must:
-
-- extract structured fields from raw invoices
-- normalize inconsistent or corrupted records
-- reconcile invoices with purchase orders
-- flag fraud and policy violations
-- investigate ambiguous cases through clarification
-- reason about upstream delivery anomalies that affect billing decisions
-
-## Environment
-
-The environment is a realistic finance-operations workspace exposed through the standard OpenEnv `reset()`, `step()`, and `state()` API.
-
-At each episode reset, the environment generates a fresh synthetic business scenario containing some combination of:
-
-- raw invoice text rendered as clean OCR, messy OCR, or adversarial OCR
-- purchase order reference data
-- approved vendor registry and historical invoice records
-- delivery logs and supply-chain records
-- ambiguous fields that require clarification turns
-
-The environment models several constraints found in real operations:
-
-- incomplete or corrupted documents
-- multiple representations of the same entity
-- conflicting sources of truth
-- deceptive lines such as fake tax or subtotal entries
-- fraud patterns that require policy grounding, not just extraction
-- limited action budget with step penalties and attempt caps
-
-This makes the world state richer than a single-document benchmark and aligns well with OpenEnv’s focus on realistic agent behavior.
-
-## Agent Capabilities
-
-The intended agent stack can be either a single frontier model or a coordinated set of specialists. A multi-agent version is especially compelling for this theme.
-
-Recommended agent roles:
-
-- Extractor Agent: converts raw text into structured invoice objects
-- Reconciliation Agent: compares invoices against purchase orders and flags mismatches
-- Audit Agent: checks vendor legitimacy, duplicate history, math fraud, and price gouging
-- Clarification Agent: decides whether a question is worth spending a turn on
-- Coordinator Agent: selects which sub-agent acts next and merges outputs into the final action
-
-Core capabilities expected from the agent system:
-
-- structured information extraction
-- entity normalization
-- cross-document comparison
-- anomaly and fraud detection
-- selective clarification and question asking
-- trajectory planning across multiple steps
-- adaptation using prior grader feedback
-
-## Tasks
-
-The current environment already contains a natural difficulty ladder plus advanced variants:
-
-1. Easy: Single Invoice Extraction
-   Extract vendor, date, currency, total, and line items from one clean invoice.
-
-2. Medium: Batch Invoice Cleaning
-   Normalize multiple messy invoices with date-format chaos, vendor typos, mixed currencies, and arithmetic inconsistencies.
-
-3. Hard: Invoice-PO Reconciliation
-   Clean invoices and compare them against purchase orders to detect overcharges, extra items, and missing items.
-
-4. Expert: Fraud Audit
-   Use policy and reference data to classify phantom vendors, duplicate submissions, price gouging, and math fraud.
-
-5. Adversarial: Trap-Resistant Extraction
-   Ignore misleading subtotal, tax, adjustment, and FX lines while recovering the true invoice structure from OCR-corrupted text.
-
-6. Negotiate: Clarification-then-Act
-   Ask targeted questions before submitting the final extraction, with a reward bonus for solving accurately using few clarifications.
-
-7. Supply Chain: Delivery Anomaly Detection
-   Identify quantity shortfalls, price spikes, unauthorized substitutions, and phantom deliveries in logistics records that affect invoice trustworthiness.
-
-These tasks support both the minimum competition requirement and a stronger story around agentic reasoning breadth.
-
-## Reward Model and Evaluation Logic
-
-The environment uses dense, trajectory-level reward rather than pure terminal success.
-
-Key evaluation properties:
-
-- scores are deterministic and normalized to `0.0–1.0`
-- reward is provided every step
-- partial credit is granted for partially correct structure and reasoning
-- repeated failure is penalized through capped attempts and end-of-episode multipliers
-- some tasks provide field-level or component-level breakdowns to support learning
-
-Illustrative reward design:
-
-- Extraction tasks score vendor, date, currency, total, and line-item accuracy separately
-- Batch tasks average per-document correctness across the batch
-- Reconciliation tasks combine extraction quality with discrepancy precision and recall
-- Fraud tasks grade both verdict quality and correct fraud-type assignment
-- Clarification tasks reward final correctness and add a bonus for solving with fewer questions
-- Supply-chain tasks reward exact and partial anomaly matching using precision and recall
-
-Why this is strong for judging:
-
-- clear task objectives
-- measurable, reproducible graders
-- meaningful partial progress signals
-- sensible episode boundaries
-- realistic penalties for wasteful or unsafe behavior
-
-## Long-Horizon and Multi-Agent Story
-
-This environment is stronger than a one-shot extraction benchmark because good performance often requires a sequence of decisions:
-
-- inspect noisy evidence
-- choose whether to normalize or reconcile first
-- decide whether ambiguity warrants clarification
-- use feedback from the last failed attempt
-- compare several external references before acting
-
-That naturally creates a long-horizon control problem. In a multi-agent implementation, the coordinator can route subtasks to specialized workers and aggregate them into a single environment action. This gives the judges a concrete example of multi-agent interaction that is useful, realistic, and measurable.
-
-## Post-Training and Self-Improvement Strategy
-
-The self-improvement loop should operate at both the environment level and the agent level.
-
-Environment-side self-improvement already fits well:
-
-- dynamic difficulty adjusts generation parameters based on recent agent performance
-- fresh synthetic scenarios prevent rote memorization
-- richer failure cases can be added over time from observed model mistakes
-
-Agent-side post-training strategy:
-
-- collect trajectories with observation, action, reward, and grader feedback
-- distill successful traces into task-specific demonstrations
-- finetune or preference-optimize the coordinator on high-reward trajectories
-- train specialist sub-agents on their own slices such as extraction, fraud, or reconciliation
-- mine common failure modes and convert them into targeted curriculum tasks
-- use reward breakdowns as auxiliary supervision for process improvement
-
-An especially strong hackathon implementation would add a memory or policy-improvement module that:
-
-- tracks recurrent failure categories per task
-- updates prompting or tool routing after each episode batch
-- increases confidence thresholds for high-risk actions
-- learns when asking a clarification question is economically justified
-
-## Why This Submission Should Score Well
-
-Against the published judging criteria, this design is well positioned because it is:
-
-- real-world: finance operations, procurement, and audit teams perform these workflows every day
-- structured: tasks are concrete, typed, and easy to evaluate automatically
-- agentic: success depends on planning, comparison, recovery from feedback, and selective questioning
-- novel enough: it goes beyond invoice OCR into reconciliation, fraud, negotiation, and supply-chain reasoning
-- extensible: more vendors, fraud patterns, policies, and document sources can be added without changing the API
-
-## Short Submission Pitch
-
-Autonomous Accounts Payable Control Tower is a real-world OpenEnv benchmark for training and evaluating AI agents on document-grounded finance operations. Agents must extract, normalize, reconcile, audit, question, and adapt across noisy multi-document workflows, receiving dense rewards and deterministic grading throughout. The environment supports single-agent and multi-agent systems, emphasizes long-horizon decision making, and includes a natural self-improvement loop through feedback-aware retries and adaptive difficulty.