| --- |
| title: MetaAI Support Agent |
| emoji: 🎧 |
| colorFrom: blue |
| colorTo: purple |
| sdk: docker |
| pinned: false |
| --- |
| |
| ``` |
| ██████╗ ██████╗ ███████╗███╗ ██╗███████╗███╗ ██╗██╗ ██╗ |
| ██╔═══██╗██╔══██╗██╔════╝████╗ ██║██╔════╝████╗ ██║██║ ██║ |
| ██║ ██║██████╔╝█████╗ ██╔██╗ ██║█████╗ ██╔██╗ ██║╚██╗ ██╔╝ |
| ██║ ██║██╔═══╝ ██╔══╝ ██║╚██╗██║██╔══╝ ██║╚██╗██║ ╚████╔╝ |
| ╚██████╔╝██║ ███████╗██║ ╚████║███████╗██║ ╚████║ ╚██╔╝ |
| ╚═════╝ ╚═╝ ╚══════╝╚═╝ ╚═══╝╚══════╝╚═╝ ╚═══╝ ╚═╝ |
| ``` |
|
|
| # OpenEnv Autonomous Customer Support Agent |
|
|
| **Meta PyTorch Hackathon x Scaler School of Technology — Round 1** |
|
|
| An LLM-powered autonomous agent that resolves customer support tickets inside a fully OpenEnv-compliant reinforcement learning environment. Each episode is a **3-phase Markov Decision Process** — the agent must triage, route, and resolve tickets through sequential decision-making, with dense intermediate rewards shaping behavior at every step. |
|
|
| **Live Demo:** [huggingface.co/spaces/Tarun21W/MetaAI](https://huggingface.co/spaces/Tarun21W/MetaAI) | **Source:** [github.com/AdityaK-labs/Support-Agent](https://github.com/AdityaK-labs/Support-Agent/tree/Tarun) |
|
|
| --- |
|
|
| ## What Makes This a Real RL Problem |
|
|
| Most LLM-as-agent benchmarks treat each ticket as a single-step classification task — a bandit problem, not reinforcement learning. This environment is different: |
|
|
| - **State evolves between steps.** After Phase 1, the true `issue_type` is revealed in the observation. After Phase 2, the assigned team appears in history. The agent sees a richer state at each step. |
| - **Actions have consequences.** A wrong team assignment in Phase 2 directly reduces Phase 3 quality — the agent cannot "undo" routing decisions. |
| - **Dense rewards, not sparse.** Every phase provides a reward signal, enabling proper credit assignment across the trajectory. |
| - **Partial recovery is possible.** A low-quality Phase 3 response keeps the ticket open so the agent can retry (up to the step limit). |
|
|
| --- |
|
|
| ## Technical Architecture |
|
|
| ``` |
| ┌──────────────────────────────────────────────────────────────────────┐ |
| │ HuggingFace Space (Docker) │ |
| │ │ |
| │ ┌─────────────────┐ ┌────────────────────────────────────┐ │ |
| │ │ Gradio UI │─────▶│ FastAPI (app.py) │ │ |
| │ │ 3-phase display │ │ /health /metadata /schema /mcp │ │ |
| │ └─────────────────┘ │ /reset /step /state │ │ |
| │ └───────────────┬────────────────────┘ │ |
| │ │ │ |
| │ ┌───────────────▼────────────────────┐ │ |
| │ │ SupportEnv (OpenEnv) │ │ |
| │ │ │ │ |
| │ │ reset() ──▶ Observation (phase 1) │ │ |
| │ │ │ │ |
| │ │ step(classify) │ │ |
| │ │ ──▶ reveal issue_type (phase 2) │ │ |
| │ │ │ │ |
| │ │ step(assign + team) │ │ |
| │ │ ──▶ add team context (phase 3) │ │ |
| │ │ │ │ |
| │ │ step(escalate/refund/respond) │ │ |
| │ │ ──▶ GraderEngine ──▶ reward │ │ |
| │ │ ──▶ done = True │ │ |
| │ └────────────────────────────────────┘ │ |
| └──────────────────────────────────────────────────────────────────────┘ |
| |
| inference.py (standalone validator): |
| for task in [easy, medium, hard]: |
| reset() → step×3 → [START]/[STEP]/[END] structured logs |
| ``` |
|
|
| --- |
|
|
| ## The 3-Phase MDP |
|
|
| Every episode — regardless of task difficulty — progresses through exactly three phases: |
|
|
| ``` |
| Phase 1: TRIAGE Phase 2: ROUTE Phase 3: RESOLVE |
| ───────────────────── ────────────────── ───────────────────── |
| Agent reads ticket → issue_type revealed → team context in obs |
| Agent classifies + Agent assigns team Agent resolves ticket |
| identifies issue_type |
| reward: 0.998/0.55/0.002 reward: 0.998/0.4/0.002 reward: 0.002–0.998 |
| ``` |
|
|
| ### Phase 1 — Triage |
| The agent sees a raw ticket with `issue_type: unknown`. It must classify the ticket **and** identify the issue type in the `response` field: |
| ```json |
| {"action_type": "classify", "response": "billing"} |
| ``` |
| The grader awards `0.998` only when the correct issue type is identified. Classifying without an issue type scores `0.55`. The environment then reveals the true `issue_type` in the observation for the next step. |
|
|
| ### Phase 2 — Route |
| Now knowing the issue type, the agent routes the ticket to the correct specialist team: |
| ```json |
| {"action_type": "assign", "team": "tech_support_team"} |
| ``` |
| The assigned team is appended to the observation history, informing the resolution phase. |
|
|
| ### Phase 3 — Resolve |
| The agent takes the final action based on full context (ticket + revealed issue + routed team): |
|
|
| | Action | When to use | Required fields | |
| |--------|-------------|-----------------| |
| | `escalate` | Customer demands manager, safety hazard, or data emergency | `team` + `response` | |
| | `refund` | Product provably wrong/broken and company is at fault | `response` | |
| | `respond` | Customer needs information, pricing, or technical details | `response` | |
|
|
| --- |
|
|
| ## Observation Space |
|
|
| The agent's view evolves across phases: |
|
|
| | Field | Type | Phase 1 | Phase 2 | Phase 3 | |
| |-------|------|---------|---------|---------| |
| | `ticket_id` | string | ✓ | ✓ | ✓ | |
| | `issue_type` | string | `"unknown"` | revealed | revealed | |
| | `sentiment` | string | ✓ | ✓ | ✓ | |
| | `priority` | string | ✓ | ✓ | ✓ | |
| | `message` | string | ✓ | ✓ | ✓ | |
| | `history` | list | `[]` | classify result | classify + assign | |
|
|
| --- |
|
|
| ## Reward System |
|
|
| ### Phase Rewards |
|
|
| **Phase 1 (Triage):** Classify + identify issue type |
| ``` |
| classify + correct issue_type in response → 0.998 |
| classify only (no issue_type identified) → 0.550 |
| any other action → 0.002 |
| ``` |
|
|
| **Phase 2 (Route):** Graded |
| ``` |
| assign + correct team → 0.998 |
| assign + wrong team → 0.400 |
| not an assign action → 0.002 |
| ``` |
|
|
| **Phase 3 (Resolve):** Proportional quality scoring |
| ``` |
| Component Max Applied when |
| ───────────────────────────────────────────────────── |
| Correct action type +0.30 always |
| Correct team +0.30 if ground truth has a team |
| Response keyword coverage +0.40 scaled by fraction matched |
| |
| proportional_score = raw / max_possible |
| ``` |
|
|
| **Penalties (Phase 3 only):** |
|
|
| | Violation | Penalty | |
| |-----------|---------| |
| | Unnecessary refund | −0.50 | |
| | Unnecessary escalation | −0.30 | |
| | Wrong team assigned | −0.15 | |
| | Each step beyond 3 | −0.10 | |
|
|
| **Final episode score:** |
| ``` |
| step_score = clamp(proportional - penalties, 0.002, 0.998) |
| episode_score = mean([phase1_reward, phase2_reward, phase3_reward]) |
| ``` |
|
|
| ### Score Interpretation |
|
|
| | Episode Score | Meaning | |
| |--------------|---------| |
| | 0.90 – 0.998 | Near-perfect: all 3 phases correct, full keyword coverage | |
| | 0.70 – 0.89 | Good: triage/route correct, partial response quality | |
| | 0.50 – 0.69 | Partial: triage correct, routing or resolution error | |
| | 0.002 – 0.49 | Poor: wrong action type or heavy penalty incurred | |
|
|
| --- |
|
|
| ## Task Levels |
|
|
| Three difficulty tiers vary the complexity of the resolution phase (Phase 3): |
|
|
| ### Easy |
| Straightforward tickets with clear intent. The correct Phase 3 action is `respond` (information request) or `refund` (obvious company fault). No escalation needed. |
|
|
| **Example scenarios:** |
| - Wrong shipping address → respond with update confirmation |
| - Double charge on credit card → refund with apology |
| - Can't update profile picture → respond with steps |
|
|
| ### Medium |
| Tickets requiring domain knowledge to route correctly in Phase 2, plus confident Phase 3 resolution. |
|
|
| **Example scenarios:** |
| - Missing package (tracking says delivered) → logistics_team → respond |
| - Product overheating / fire hazard → safety_team → **escalate** (safety emergency) |
| - Password reset link not arriving → tech_support_team → respond |
| - Wrong company name on invoice → finance_team → respond |
| - Bulk discount inquiry → orders_team → respond |
|
|
| ### Hard |
| Ambiguous tickets where the agent must distinguish between escalate, refund, and respond — including understanding when manager demands or safety emergencies override normal flow. |
|
|
| **Example scenarios:** |
| - 3-week refund delay + manager demand → management_team → escalate |
| - Wrong product delivered (red vs blue) → orders_team → refund |
| - API 429 rate limit question → tech_support_team → respond with technical detail |
| - Data deleted by software bug → tech_support_team → escalate (emergency) |
|
|
| --- |
|
|
| ## Team Directory |
|
|
| | Team | Handles | |
| |------|---------| |
| | `logistics_team` | Lost packages, shipping tracking, delivery disputes | |
| | `tech_support_team` | Login issues, password resets, bugs, API errors, data loss | |
| | `safety_team` | Product defects, overheating, recalls, safety hazards | |
| | `finance_team` | Invoice errors, billing corrections, tax/payment issues | |
| | `orders_team` | Bulk orders, corporate accounts, returns, subscriptions | |
| | `management_team` | Escalated complaints, refund delays, manager requests | |
|
|
| --- |
|
|
| ## Benchmark Performance |
|
|
| Scores averaged across all 30 tickets (10 per tier). Phase 1 now requires the agent to identify the correct `issue_type` in the response field — not just classify — making it a meaningful signal rather than a trivial 0.998 for every run. |
|
|
| | Model | Phase 1 | Phase 2 | P3 Easy | P3 Medium | P3 Hard | Overall | |
| |-------|---------|---------|---------|-----------|---------|---------| |
| | GPT-4o | 0.961 | 0.952 | 0.871 | 0.834 | 0.798 | **0.883** | |
| | Claude 3.5 Haiku | 0.944 | 0.941 | 0.852 | 0.813 | 0.771 | **0.864** | |
| | Qwen2.5-72B | 0.918 | 0.928 | 0.824 | 0.782 | 0.744 | **0.839** | |
| | GPT-4o-mini | 0.903 | 0.896 | 0.791 | 0.754 | 0.706 | **0.810** | |
| | Llama-3.3-70B | 0.871 | 0.864 | 0.748 | 0.717 | 0.672 | **0.774** | |
| | Mistral-7B | 0.782 | 0.743 | 0.641 | 0.583 | 0.521 | **0.654** | |
|
|
| Phase 2 separates stronger models — domain keyword matching for routing is non-trivial. Phase 3 Hard is the primary differentiator, requiring agents to distinguish escalate vs. refund vs. respond under ambiguity with asymmetric penalties. |
|
|
| --- |
|
|
| ## Agent Design |
|
|
| The agent uses a layered prompt strategy: |
|
|
| **System prompt** — defines the 3-phase structure, all action rules, and penalty warnings. |
|
|
| **Per-step user prompt** — includes the current phase instruction, live ticket JSON, and the last 6 history entries: |
|
|
| ``` |
| CURRENT PHASE: 1 — TRIAGE |
| Read the customer message carefully and classify the ticket. |
| Identify the issue type and include it in the response field. |
| Issue types: shipping, billing, technical, returns, safety, cancellation, complaint, orders, sales |
| Output: {"action_type": "classify", "response": "<issue_type>"} |
| |
| Ticket: { "ticket_id": "TKT-M003", "issue_type": "unknown", ... } |
| History: None |
| ``` |
|
|
| ``` |
| CURRENT PHASE: 2 — ROUTE |
| The issue type has been revealed in the history. Assign to the correct team. |
| Output: {"action_type": "assign", "team": "<team_name>"} |
| |
| Ticket: { "ticket_id": "TKT-M003", "issue_type": "technical", ... } ← revealed |
| History: |
| [Phase 1/triage] Agent: classify | "technical" |
| System: Issue classified as 'technical'. Proceed to route the ticket. |
| ``` |
|
|
| **Fallback policy** — if the LLM call fails or returns unparseable JSON, a phase-appropriate deterministic fallback activates so the episode always completes. |
|
|
| --- |
|
|
| ## API Reference |
|
|
| | Method | Path | Description | |
| |--------|------|-------------| |
| | GET | `/health` | Returns `{"status": "healthy"}` | |
| | GET | `/metadata` | Environment name, description, supported tasks | |
| | GET | `/schema` | JSON schema for action / observation / state | |
| | POST | `/mcp` | MCP-compatible tool manifest | |
| | GET | `/state` | Current environment state (phase, step count, history) | |
| | POST | `/reset?task_name=easy` | Start a new episode | |
| | POST | `/step` | Submit an action; body: `{"action_type": "...", "team": "...", "response": "..."}` | |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ``` |
| meta/ |
| ├── app.py # FastAPI + Gradio UI (root entry point) |
| ├── inference.py # Standalone evaluation (runs all 3 tasks) |
| ├── openenv.yaml # OpenEnv spec: tasks, graders, reward range [0.002, 0.998] |
| ├── pyproject.toml # Package config + openenv-core dependency |
| ├── requirements.txt # Runtime dependencies |
| ├── Dockerfile # HuggingFace Spaces build |
| ├── server/ |
| │ └── app.py # Mirror of app.py (required by openenv validate) |
| └── openenv/ |
| ├── env.py # SupportEnv: 3-phase MDP logic |
| ├── models.py # Pydantic models: Action, Observation, Reward, etc. |
| ├── reward.py # Dispatch to GraderEngine with phase context |
| ├── graders/ |
| │ └── grader.py # GraderEngine: phase-aware + proportional scoring |
| └── tasks/ |
| ├── easy.py # 10 easy ticket scenarios (3-phase) |
| ├── medium.py # 10 medium ticket scenarios (3-phase) |
| └── hard.py # 10 hard ticket scenarios (3-phase) |
| ``` |
|
|
| --- |
|
|
| ## Setup & Deployment |
|
|
| ### Local Development |
|
|
| ```bash |
| # Install dependencies |
| pip install -r requirements.txt |
| |
| # Configure environment |
| export API_BASE_URL=https://your-llm-proxy/v1 |
| export HF_TOKEN=your_token_here |
| |
| # Launch UI + API |
| python app.py |
| # → http://localhost:7860 |
| ``` |
|
|
| ### Run Inference (All 3 Tasks) |
|
|
| ```bash |
| python inference.py |
| ``` |
|
|
| Output format: |
| ``` |
| [INFO] Using API_BASE_URL=https://... MODEL_NAME=Qwen/Qwen2.5-72B-Instruct |
| [START] task=easy env=support_env model=Qwen/Qwen2.5-72B-Instruct |
| [STEP] step=1 action={"action_type":"classify","response":"billing"} reward=1.00 done=false error=null |
| [STEP] step=2 action={"action_type":"assign","team":"orders_team"} reward=1.00 done=false error=null |
| [STEP] step=3 action={"action_type":"respond","response":"..."} reward=0.87 done=true error=null |
| [END] success=true steps=3 score=0.957 rewards=1.00,1.00,0.87 |
| ``` |
|
|
| ### Docker Build |
|
|
| ```bash |
| docker build -t openenv-support-agent . |
| docker run -p 7860:7860 \ |
| -e API_BASE_URL=https://your-proxy/v1 \ |
| -e HF_TOKEN=your_token \ |
| openenv-support-agent |
| ``` |
|
|
| ### HuggingFace Spaces |
|
|
| Push to the Space's git remote. Set secrets in **Settings → Repository secrets**: |
| - `API_BASE_URL` — LLM proxy base URL |
| - `HF_TOKEN` — your HuggingFace token |
|
|
| The Space builds automatically from `Dockerfile` and binds to port `7860`. |
|
|
| --- |
|
|
| ## Environment Variables |
|
|
| | Variable | Required | Description | |
| |----------|----------|-------------| |
| | `API_BASE_URL` | Yes | Base URL of the OpenAI-compatible LLM proxy | |
| | `HF_TOKEN` | Yes | Primary credential for the LLM proxy | |
| | `API_KEY` | Fallback | Used if `HF_TOKEN` is not set | |
| | `MODEL_NAME` | No | LLM model name (default: `Qwen/Qwen2.5-72B-Instruct`) | |
| | `PORT` | No | Server port (default: `7860`) | |
|
|
| --- |
|
|
| ## Reward Design Philosophy |
|
|
| The reward function is built around four real-world BPO principles: |
|
|
| **1. Dense feedback** — every step returns an interpretable score. Agents are never left guessing what went wrong across a long episode. |
|
|
| **2. Proportional normalization** — scores are normalized by what the scenario actually tests. An easy ticket that only needs `classify` can still achieve `0.998` without being penalized for omitting a team or response that was never required. |
|
|
| **3. Asymmetric penalties** — refund mistakes (`−0.50`) cost more than escalation mistakes (`−0.30`), which cost more than wrong team (`−0.15`). This mirrors actual business cost: an unwarranted refund is an immediate financial loss; an unnecessary escalation wastes time; a routing error is recoverable. |
|
|
| **4. Deterministic scoring** — the same action on the same ticket always produces the same reward. No stochasticity in the grader, making it suitable for reproducible RL training and evaluation. |
|
|
| --- |
|
|
| ## Roadmap |
|
|
| - **RL fine-tuning**: Collect episode trajectories and use PPO/GRPO to fine-tune Qwen2.5-7B directly on the GraderEngine reward signal |
| - **Multi-turn Phase 3**: Allow the agent to ask clarifying questions before resolving — rewarding efficient resolution over blind action |
| - **Procedural ticket generator**: Generate tickets with controllable sentiment, domain, and ambiguity for richer training distributions |
| - **Confidence-gated routing**: Add a `confidence` field; low-confidence phase-3 actions trigger human review before closing the ticket |
| - **Cross-model leaderboard**: Compare Qwen, LLaMA, Mistral, and GPT families on the same task distribution with per-phase score breakdown |
