Spaces:
Sleeping
Sleeping
Update README with comprehensive Human Validation Agent documentation
Browse files
README.md
CHANGED
|
@@ -7,7 +7,6 @@ sdk: gradio
|
|
| 7 |
pinned: false
|
| 8 |
short_description: A simple Gradio app that greets users by name
|
| 9 |
---
|
| 10 |
-
|
| 11 |
# StrategyStack Recovery/Enrichment Agent
|
| 12 |
|
| 13 |
## Overview
|
|
@@ -19,15 +18,208 @@ This repository contains a modular recovery/enrichment agent capable of prioriti
|
|
| 19 |
- Activity logging for all agent actions, outcome, errors, and escalations in [`activity_log.csv`](./activity_log.csv).
|
| 20 |
- Modular integration: Upstream and downstream agent handoff (query_search_agent, validation_agent, etc.) supported for live flows.
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
## Usage
|
|
|
|
| 23 |
1. **Configure enrichment sources:**
|
| 24 |
Edit `docs/configs/enrichment_config.json` to set source order, rate limits, retry policies, and handoff rules.
|
|
|
|
| 25 |
2. **Run simulation or queue handoff:**
|
| 26 |
Execute `docs/enrichment_handoff_simulation.py` to simulate a queue item enrichment with browser fallback and human escalation.
|
| 27 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
Inspect `activity_log.csv` for all enrichment, handoff, error, and escalation events (timestamped and detailed).
|
| 29 |
|
| 30 |
## Example Config
|
|
|
|
| 31 |
```json
|
| 32 |
{
|
| 33 |
"enrichment_sources": {
|
|
@@ -44,6 +236,7 @@ This repository contains a modular recovery/enrichment agent capable of prioriti
|
|
| 44 |
```
|
| 45 |
|
| 46 |
## Live Integration
|
|
|
|
| 47 |
- **Upstream:** Receives items from ingest/query queue.
|
| 48 |
- **Enrichment:** Prioritized API calls with fallback if rates/quality unmet.
|
| 49 |
- **Browser Agent:** Triggered if API unavailable, rate-limited, or data quality insufficient.
|
|
@@ -51,8 +244,9 @@ This repository contains a modular recovery/enrichment agent capable of prioriti
|
|
| 51 |
- **Audit Trail:** Complete log of actions, fallback, escalations in CSV.
|
| 52 |
|
| 53 |
## Preparing for Next Agent
|
|
|
|
| 54 |
- Extend `docs/configs/enrichment_config.json` to add sources or escalation criteria.
|
| 55 |
- Integrate additional downstream agents via `integration` section of config.
|
| 56 |
- Full audit log available for compliance/modules handoff.
|
| 57 |
|
| 58 |
-
---
|
|
|
|
| 7 |
pinned: false
|
| 8 |
short_description: A simple Gradio app that greets users by name
|
| 9 |
---
|
|
|
|
| 10 |
# StrategyStack Recovery/Enrichment Agent
|
| 11 |
|
| 12 |
## Overview
|
|
|
|
| 18 |
- Activity logging for all agent actions, outcome, errors, and escalations in [`activity_log.csv`](./activity_log.csv).
|
| 19 |
- Modular integration: Upstream and downstream agent handoff (query_search_agent, validation_agent, etc.) supported for live flows.
|
| 20 |
|
| 21 |
+
## Human Validation Agent Module
|
| 22 |
+
|
| 23 |
+
The Human Validation Agent module provides comprehensive functionality for manual review and validation of leads that require human intervention. This module is fully integrated with the lead queue system and provides robust validation, feedback, and handoff capabilities.
|
| 24 |
+
|
| 25 |
+
### Key Features
|
| 26 |
+
|
| 27 |
+
1. **Queue Integration**
|
| 28 |
+
- Automatically pulls leads marked with `needs_human_review` flag from the queue
|
| 29 |
+
- Supports configurable batch sizes for efficient review workflows
|
| 30 |
+
- Filters by status (`pending_review`, `flagged`) for targeted processing
|
| 31 |
+
|
| 32 |
+
2. **Batch Review Interface**
|
| 33 |
+
- Process multiple leads in a single batch operation
|
| 34 |
+
- Four resolution options:
|
| 35 |
+
- **Approve**: Move lead to approved queue for downstream processing
|
| 36 |
+
- **Edit**: Update lead data and move to approved queue
|
| 37 |
+
- **Hold**: Keep in review queue with hold status for retry
|
| 38 |
+
- **Scrub**: Archive lead with documented reason
|
| 39 |
+
|
| 40 |
+
3. **Schema Validation**
|
| 41 |
+
- Pydantic-based schema validation for all review actions
|
| 42 |
+
- Business rule validation (e.g., required fields for edit actions)
|
| 43 |
+
- Reason validation for hold and scrub actions (minimum 10 characters)
|
| 44 |
+
- Real-time validation feedback with user-friendly error messages
|
| 45 |
+
|
| 46 |
+
4. **Feedback & Guardrails**
|
| 47 |
+
- Invalid actions are rejected with clear, actionable feedback
|
| 48 |
+
- Comprehensive error handling and logging
|
| 49 |
+
- Validation error tracking for audit purposes
|
| 50 |
+
- Success rate statistics and review analytics
|
| 51 |
+
|
| 52 |
+
5. **Resolution & Handoff**
|
| 53 |
+
- **Approved leads**: Routed to `approved_leads` queue for downstream processing
|
| 54 |
+
- **Edited leads**: Updated data validated and routed to `approved_leads` queue
|
| 55 |
+
- **Held leads**: Maintained in `on_hold` queue with documented reason
|
| 56 |
+
- **Scrubbed leads**: Archived to `scrubbed_leads` queue with reason
|
| 57 |
+
|
| 58 |
+
### API Interface
|
| 59 |
+
|
| 60 |
+
```python
|
| 61 |
+
from human_validation_agent import HumanValidationAgent, create_review_interface
|
| 62 |
+
|
| 63 |
+
# Initialize agent
|
| 64 |
+
agent = HumanValidationAgent(queue_connection=your_queue_connection)
|
| 65 |
+
|
| 66 |
+
# Create interface
|
| 67 |
+
interface = create_review_interface(agent)
|
| 68 |
+
|
| 69 |
+
# Get pending reviews
|
| 70 |
+
pending_leads = interface["get_pending_reviews"](limit=50)
|
| 71 |
+
|
| 72 |
+
# Submit single review
|
| 73 |
+
review_result = interface["submit_review"]({
|
| 74 |
+
"lead_id": "lead_001",
|
| 75 |
+
"action": "approve",
|
| 76 |
+
"reviewer_id": "reviewer_123"
|
| 77 |
+
})
|
| 78 |
+
|
| 79 |
+
# Submit batch review
|
| 80 |
+
batch_result = interface["submit_batch_review"]({
|
| 81 |
+
"reviews": [
|
| 82 |
+
{
|
| 83 |
+
"lead_id": "lead_001",
|
| 84 |
+
"action": "approve",
|
| 85 |
+
"reviewer_id": "reviewer_123"
|
| 86 |
+
},
|
| 87 |
+
{
|
| 88 |
+
"lead_id": "lead_002",
|
| 89 |
+
"action": "edit",
|
| 90 |
+
"edited_data": {
|
| 91 |
+
"name": "Updated Name",
|
| 92 |
+
"email": "updated@email.com",
|
| 93 |
+
"company": "Updated Corp"
|
| 94 |
+
},
|
| 95 |
+
"reviewer_id": "reviewer_123"
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"lead_id": "lead_003",
|
| 99 |
+
"action": "scrub",
|
| 100 |
+
"reason": "Lead is a competitor and not a valid prospect",
|
| 101 |
+
"reviewer_id": "reviewer_123"
|
| 102 |
+
}
|
| 103 |
+
]
|
| 104 |
+
})
|
| 105 |
+
|
| 106 |
+
# Get statistics
|
| 107 |
+
stats = interface["get_stats"]()
|
| 108 |
+
|
| 109 |
+
# Export history
|
| 110 |
+
interface["export_history"]("review_history.json")
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
### Schema Definitions
|
| 114 |
+
|
| 115 |
+
#### LeadReview Schema
|
| 116 |
+
```python
|
| 117 |
+
{
|
| 118 |
+
"lead_id": str, # Required: Unique lead identifier
|
| 119 |
+
"action": str, # Required: "approve", "edit", "hold", or "scrub"
|
| 120 |
+
"reason": str | None, # Required for "hold" and "scrub" actions
|
| 121 |
+
"edited_data": dict | None, # Required for "edit" action
|
| 122 |
+
"reviewer_id": str, # Required: Reviewer identifier
|
| 123 |
+
"reviewed_at": datetime # Auto-generated timestamp
|
| 124 |
+
}
|
| 125 |
+
```
|
| 126 |
+
|
| 127 |
+
#### BatchReviewRequest Schema
|
| 128 |
+
```python
|
| 129 |
+
{
|
| 130 |
+
"reviews": List[LeadReview], # Required: List of review objects
|
| 131 |
+
"batch_id": str | None # Optional: Batch identifier
|
| 132 |
+
}
|
| 133 |
+
```
|
| 134 |
+
|
| 135 |
+
### Validation Rules
|
| 136 |
+
|
| 137 |
+
1. **Approve Action**: No additional fields required
|
| 138 |
+
2. **Edit Action**: Must include `edited_data` with required fields (`name`, `email`, `company`)
|
| 139 |
+
3. **Hold Action**: Must include `reason` (minimum 10 characters)
|
| 140 |
+
4. **Scrub Action**: Must include `reason` (minimum 10 characters)
|
| 141 |
+
|
| 142 |
+
### Response Format
|
| 143 |
+
|
| 144 |
+
#### Batch Review Response
|
| 145 |
+
```json
|
| 146 |
+
{
|
| 147 |
+
"batch_id": "batch_1730832123.456",
|
| 148 |
+
"total_reviews": 3,
|
| 149 |
+
"successful": 2,
|
| 150 |
+
"failed": 1,
|
| 151 |
+
"results": [
|
| 152 |
+
{
|
| 153 |
+
"lead_id": "lead_001",
|
| 154 |
+
"action": "approve",
|
| 155 |
+
"status": "approved",
|
| 156 |
+
"next_queue": "approved_leads",
|
| 157 |
+
"timestamp": "2025-11-05T13:23:00"
|
| 158 |
+
}
|
| 159 |
+
],
|
| 160 |
+
"errors": [
|
| 161 |
+
{
|
| 162 |
+
"lead_id": "lead_002",
|
| 163 |
+
"error": "Missing required field in edited_data: email",
|
| 164 |
+
"feedback": "Edited lead data must include all required fields: name, email, and company."
|
| 165 |
+
}
|
| 166 |
+
]
|
| 167 |
+
}
|
| 168 |
+
```
|
| 169 |
+
|
| 170 |
+
### Statistics & Reporting
|
| 171 |
+
|
| 172 |
+
The agent tracks comprehensive statistics:
|
| 173 |
+
- Total reviews processed
|
| 174 |
+
- Reviews by action type (approve, edit, hold, scrub)
|
| 175 |
+
- Validation error count
|
| 176 |
+
- Success rate calculation
|
| 177 |
+
- Full history export to JSON
|
| 178 |
+
|
| 179 |
+
### Integration Points
|
| 180 |
+
|
| 181 |
+
1. **Upstream**: Receives leads from enrichment agent or browser agent
|
| 182 |
+
2. **Queue Connection**: Queries and updates lead queue system
|
| 183 |
+
3. **Downstream**: Routes to appropriate queues based on action:
|
| 184 |
+
- `approved_leads` → Next processing stage
|
| 185 |
+
- `on_hold` → Temporary hold for retry
|
| 186 |
+
- `scrubbed_leads` → Archived/removed from pipeline
|
| 187 |
+
|
| 188 |
+
### Error Handling
|
| 189 |
+
|
| 190 |
+
- Schema validation errors return detailed feedback
|
| 191 |
+
- Business rule violations provide actionable guidance
|
| 192 |
+
- All errors logged with timestamps for audit
|
| 193 |
+
- Failed reviews don't block successful reviews in batch operations
|
| 194 |
+
|
| 195 |
+
### Audit Trail
|
| 196 |
+
|
| 197 |
+
All review activities are logged with:
|
| 198 |
+
- Lead ID and action taken
|
| 199 |
+
- Reviewer ID and timestamp
|
| 200 |
+
- Reason for hold/scrub actions
|
| 201 |
+
- Updated fields for edit actions
|
| 202 |
+
- Validation errors and feedback
|
| 203 |
+
- Export capability for compliance reporting
|
| 204 |
+
|
| 205 |
+
---
|
| 206 |
+
|
| 207 |
## Usage
|
| 208 |
+
|
| 209 |
1. **Configure enrichment sources:**
|
| 210 |
Edit `docs/configs/enrichment_config.json` to set source order, rate limits, retry policies, and handoff rules.
|
| 211 |
+
|
| 212 |
2. **Run simulation or queue handoff:**
|
| 213 |
Execute `docs/enrichment_handoff_simulation.py` to simulate a queue item enrichment with browser fallback and human escalation.
|
| 214 |
+
|
| 215 |
+
3. **Human validation workflow:**
|
| 216 |
+
Use `human_validation_agent.py` to pull and review leads requiring manual intervention.
|
| 217 |
+
|
| 218 |
+
4. **Audit log review:**
|
| 219 |
Inspect `activity_log.csv` for all enrichment, handoff, error, and escalation events (timestamped and detailed).
|
| 220 |
|
| 221 |
## Example Config
|
| 222 |
+
|
| 223 |
```json
|
| 224 |
{
|
| 225 |
"enrichment_sources": {
|
|
|
|
| 236 |
```
|
| 237 |
|
| 238 |
## Live Integration
|
| 239 |
+
|
| 240 |
- **Upstream:** Receives items from ingest/query queue.
|
| 241 |
- **Enrichment:** Prioritized API calls with fallback if rates/quality unmet.
|
| 242 |
- **Browser Agent:** Triggered if API unavailable, rate-limited, or data quality insufficient.
|
|
|
|
| 244 |
- **Audit Trail:** Complete log of actions, fallback, escalations in CSV.
|
| 245 |
|
| 246 |
## Preparing for Next Agent
|
| 247 |
+
|
| 248 |
- Extend `docs/configs/enrichment_config.json` to add sources or escalation criteria.
|
| 249 |
- Integrate additional downstream agents via `integration` section of config.
|
| 250 |
- Full audit log available for compliance/modules handoff.
|
| 251 |
|
| 252 |
+
---
|