document-classification-env / HOW_IT_WORKS.md
TanujInsane
initial commit
8ddfad6
|
Raw History Blame Contribute Delete
15 kB

🎯 HOW THE OPENENV ENVIRONMENT WORKS

Executive Overview

This is a document classification and routing environment where AI agents learn to classify incoming documents and route them to the correct department. It's like a smart mail sorter that learns to categorize letters!


πŸ—οΈ Architecture Diagram

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    AGENT INTERACTION LOOP                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚
                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  1. Agent receives OBSERVATION      β”‚
    β”‚     (document content + features)   β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚
                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  2. Agent makes DECISION            β”‚
    β”‚     (picks a category)              β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚
                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  3. Environment steps (REWARD)      β”‚
    β”‚     β€’ Accuracy bonus/penalty        β”‚
    β”‚     β€’ Speed bonus                   β”‚
    β”‚     β€’ Episode progress              β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚
                              β–Ό
    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
    β”‚  4. Repeat until episode ends       β”‚
    β”‚     (all documents classified)      β”‚
    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ“‹ Complete Workflow

Phase 1: Initialize Environment

from environment import DocumentClassificationEnv

# Create an easy environment
env = DocumentClassificationEnv(task_difficulty="easy")

# Returns:
# - Action space: Discrete(5) = 5 categories
# - Observation space: Dict with document features

Phase 2: Reset for Episode

observation, info = env.reset()

# Returns observation dict:
# {
#   "document_id": "doc_000000",
#   "content": "My invoice shows an incorrect amount...",
#   "word_count": [45],
#   "has_urgency_markers": [0],
#   "features": [0.12, -0.34, 0.56, ...],  # 100-dim vector
#   "document_index": [0],
#   "total_documents": [100]
# }

Phase 3: Agent Decision

# Agent analyzes the document
action = agent.decide(observation)
# Returns: 1 (Billing category)

Phase 4: Environment Steps

obs, reward, done, truncated, info = env.step(action)

# Returns:
# - obs: Next document observation
# - reward: 1.0 (correct) or -0.5 (incorrect) + speed bonus
# - done: False (more docs) or True (episode complete)
# - info: {
#     "is_correct": True,
#     "true_category": "Billing",
#     "predicted_category": "Billing",
#     "episode_accuracy": 0.75,
#     "processing_time_ms": 45.2
#   }

Phase 5: Episode Summary

When done=True, info includes:
{
  "episode_summary": {
    "total_reward": 75.3,
    "accuracy": 0.87,
    "average_reward": 0.873,
    "average_processing_time_ms": 50.1,
    "total_episode_time_seconds": 5.2
  }
}

🎲 Task Details

Easy Task

Goal: Classify 100 documents into 5 categories
Categories: [General, Billing, Support, Technical, HR]
Challenge: Low - documents are pre-processed
Time Limit: None
Baseline Score: 0.78
Perfect Score: 1.0

Example Document:

Content: "I need to update my billing address."
True Label: Billing (category index 1)
Features: 100-dimensional TF-IDF vector
Reward if correct: +1.0
Reward if incorrect: -0.5

Medium Task

Goal: Classify 500 documents into 10 categories
Categories: General, Billing, Billing-Dispute, Support, Technical,
           Technical-Bug, HR-Payroll, HR-Benefits, Legal, Executive
Challenge: Medium - more categories, time constraints
Time Limit: 2 seconds per decision
Baseline Score: 0.65

Hard Task

Goal: Classify 1000 documents into 20 categories
Challenge: High - fine-grained, speed-accuracy tradeoff
Time Limit: 1 second per decision
Baseline Score: 0.52
Speed-Accuracy Tradeoff: Agents must balance speed vs accuracy

🧠 How Features Work

Feature Extraction

# Text β†’ TF-IDF Vector (100 dimensions)
document = "I was overcharged on my last invoice"
features = tfidf_vectorizer.transform([document]).toarray()
# Output: [0.45, -0.23, 0.67, 0.12, ..., -0.34]  (100 values)

Feature Interpretation

Positive values: Word appears frequently and is important
Negative values: Word is common (low importance - IDF weighted)
Magnitude: How important this word is to the document

Example:
  "invoice" β†’ high positive (rare, important word)
  "the" β†’ near zero (common, low importance)
  "billing" β†’ very high positive (rare AND diagnostic)

πŸ’° Reward Function

Accuracy Component

If agent classifies CORRECTLY:
  reward += 1.0  βœ“

If agent classifies INCORRECTLY:
  reward -= 0.5  βœ—

Speed Bonus Component

Easy Task:
  Process < 100ms β†’ reward += 0.1

Medium Task:
  Process < 200ms β†’ reward += 0.15
  Process < 500ms β†’ reward += 0.05

Hard Task (1s limit):
  Process < 100ms β†’ reward += 0.2
  Process < 300ms β†’ reward += 0.1

Total Reward Example

Correct + Fast (medium):
  reward = 1.0 (correct) + 0.15 (speed) = 1.15

Incorrect + Fast (medium):
  reward = -0.5 (incorrect) + 0.15 (speed) = -0.35

Correct + Slow (hard):
  reward = 1.0 (correct) + 0.0 (too slow) = 1.0

πŸ“Š Episode Flow

Episode Start (100 documents for Easy)
β”‚
β”œβ”€ Doc 1: "My billing address..."
β”‚  β”œβ”€ Agent predicts: Billing βœ“
β”‚  β”œβ”€ Reward: +1.0 + 0.1 (speed) = +1.1
β”‚  └─ Accuracy: 1/1 = 100%
β”‚
β”œβ”€ Doc 2: "I found a bug..."
β”‚  β”œβ”€ Agent predicts: Support βœ— (should be Technical)
β”‚  β”œβ”€ Reward: -0.5 + 0.1 (speed) = -0.4
β”‚  └─ Accuracy: 1/2 = 50%
β”‚
β”œβ”€ Doc 3: "Can I file a complaint?"
β”‚  β”œβ”€ Agent predicts: HR βœ“
β”‚  β”œβ”€ Reward: +1.0 + 0.1 (speed) = +1.1
β”‚  └─ Accuracy: 2/3 = 67%
β”‚
...
β”‚
└─ Doc 100: (last document)
   β”œβ”€ Agent predicts: General βœ“
   β”œβ”€ Reward: +1.0
   └─ EPISODE COMPLETE
      Total Reward: 87.3
      Final Accuracy: 87%
      Avg Time: 50ms

πŸ€– Agent Decision Process

Simple Baseline Agent (Rule-Based)

def baseline_agent(observation):
    content = observation['content'].lower()
    
    # Check for keywords
    if "billing" in content or "invoice" in content:
        return 1  # Billing category
    elif "bug" in content or "error" in content:
        return 3  # Technical category
    elif "payroll" in content or "salary" in content:
        return 4  # HR category
    else:
        return 0  # Default: General

Advanced ML Agent (Trained)

def ml_agent(observation):
    # Use pre-trained neural network
    features = torch.tensor(observation['features'])
    logits = model(features)
    action = torch.argmax(logits).item()
    return action

πŸ… Grading System

How Agents Are Scored

Grading Process:
1. Run agent on test set (100/500/1000 documents)
2. Collect predictions and ground truths
3. Calculate metrics:
   - Accuracy = correct / total
   - Precision = TP / (TP + FP) per category
   - Recall = TP / (TP + FN) per category
   - F1 Score = harmonic mean(precision, recall)
4. Scale to difficulty level
5. Return final score (0.0 - 1.0)

Difficulty-Aware Scoring

EASY:
  score = accuracy
  (No speed penalty)

MEDIUM:
  score = 0.8 Γ— accuracy + 0.2 Γ— speed_bonus
  (Balanced approach)

HARD:
  score = 0.75 Γ— accuracy + 0.25 Γ— speed_bonus
  (Emphasizes accuracy, rewards speed)

Example Grading

Agent Performance on Easy:
β”œβ”€ Correct: 87/100 documents
β”œβ”€ Accuracy: 0.87
β”œβ”€ Avg Speed: 45ms
β”œβ”€ Speed Bonus: 0.1 (< 100ms)
└─ Final Score: 0.87

Agent Performance on Hard:
β”œβ”€ Correct: 730/1000 documents
β”œβ”€ Accuracy: 0.73
β”œβ”€ Avg Speed: 95ms
β”œβ”€ Speed Bonus: 0.2 (< 100ms)
└─ Final Score: 0.75 Γ— 0.73 + 0.25 Γ— 0.2 = 0.595

πŸ”„ State API

Accessing Environment State

state = env.state()

# Returns:
{
  "current_observation": {...},  # Current document
  "current_document_index": 47,  # Which document we're on
  "total_documents": 100,        # Total in episode
  "episode_reward_total": 42.7,  # Sum of all rewards
  "episode_accuracy": 0.894,     # Accuracy so far
  "average_processing_time_ms": 48.3,
  "task_difficulty": "easy"
}

πŸ“ˆ Training an Agent

Learning Curve Example

Episode 1: Score 0.35 (Random decisions)
Episode 5: Score 0.52 (Learning keywords)
Episode 10: Score 0.68 (Pattern recognition)
Episode 20: Score 0.79 (Specialized learning)
Episode 50: Score 0.91 (Near optimal)

πŸ” Reproducibility

Seed Control

# Same seed = identical data and scores
env1 = DocumentClassificationEnv("easy", seed=42)
env2 = DocumentClassificationEnv("easy", seed=42)

obs1, _ = env1.reset(seed=42)
obs2, _ = env2.reset(seed=42)

# obs1['features'] == obs2['features']  βœ“

Fixed Test Sets

Test Set is FIXED:
- Easy: Same 100 documents every evaluation
- Medium: Same 500 documents every evaluation
- Hard: Same 1000 documents every evaluation

This ensures:
βœ“ Agent scores are comparable across runs
βœ“ Different agents can be fairly compared
βœ“ Reproducible baseline metrics

🎯 Categories Explained

Easy (5 categories)

0: General        β†’ Default/uncategorized
1: Billing        β†’ Payment, invoice, charges
2: Support        β†’ Help, assistance, how-to
3: Technical      β†’ Technical issues, bugs
4: HR             β†’ HR, payroll, benefits

Medium (10 categories)

0: General
1: Billing        β†’ Payments
2: Billing-Dispute β†’ Overcharge disputes
3: Support        β†’ General support
4: Technical      β†’ General technical
5: Technical-Bug  β†’ Specific bugs/errors
6: HR-Payroll     β†’ Salary/payroll
7: HR-Benefits    β†’ Benefits/insurance
8: Legal          β†’ Legal matters
9: Executive      β†’ Management level

Hard (20 categories)

0-3: Billing variants
4-5: Support urgent/normal
6-8: Technical variants
9-11: HR variants
12-14: Legal variants
15-16: Executive variants
17: Finance
18: Marketing
19: Operations

πŸ’‘ Real-World Use Case

A company receives 1000 emails/documents per day:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Email comes in:     β”‚
β”‚ "I was charged     β”‚
β”‚ twice for order    β”‚
β”‚ #5432"             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Environment processes:      β”‚
β”‚ - Extract features (TF-IDF) β”‚
β”‚ - Normalize                 β”‚
β”‚ - Present to agent          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Agent decides: Category 2   β”‚
β”‚ (Billing-Dispute)          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Route to:                   β”‚
β”‚ Billing Department          β”‚
β”‚ Dispute Resolution Team     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Get reward:                 β”‚
β”‚ +1.0 (correct routing)      β”‚
β”‚ Learn from this decision    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸš€ Running the Environment

Local Execution

pip install -r requirements.txt
python test_environment.py          # Verify everything works
python baseline_inference.py        # See baseline performance
python app.py                       # Interactive web interface

Docker Execution

docker build -t doc-classifier .
docker run -p 7860:7860 doc-classifier
# Access at http://localhost:7860

πŸ“Š Performance Interpretation

Baseline Scores

Easy: 0.78
β”œβ”€ Simple keyword matching strategy
β”œβ”€ Achievable by any RL agent with basic features
└─ Gap to perfect (1.0) = 0.22 (22%)

Medium: 0.65
β”œβ”€ Extended keywords + category variants
β”œβ”€ Requires learning category distinctions
└─ Gap to target (0.85) = 0.20 (20%)

Hard: 0.52
β”œβ”€ Limited by heuristic approach
β”œβ”€ Needs ML model for fine-grained classification
└─ Gap to target (0.75) = 0.23 (23%)

What Scores Mean

0.95+: Expert-level performance
0.85+: Strong RL agent
0.75+: Well-trained model
0.65+: Basic learning
0.50+: Better than random
0.00 : Completely random guessing

✨ Key Insights

  1. Real-World Problem: Document routing is a genuine business challenge
  2. Progressive Difficulty: Tasks scale from simple to complex
  3. Speed-Accuracy Tradeoff: Hard task forces intelligent decisions
  4. Reproducible: Same seed = same results every time
  5. Deterministic Grading: Fair comparison between agents
  6. Extensible: Easy to add custom agents

πŸŽ“ Summary

The environment teaches agents to: βœ… Understand document content (NLP) βœ… Extract meaningful patterns (feature importance) βœ… Make fast decisions (speed optimization) βœ… Balance accuracy vs speed (tradeoff learning) βœ… Generalize to new documents (test set performance)

This mirrors real-world AI deployment challenges!


Ready to try it? Start with:

pip install -r requirements.txt
python test_environment.py
python app.py

Then explore the code and try building your own agent!