Spaces:
Sleeping
Download codebase_explanation.md from harshitmahour360/ARY: direct link, hf CLI and curl.
- Browser
- Download file 5.89 kB
-
https://huggingface.co/spaces/harshitmahour360/ARY/resolve/main/codebase_explanation.md
- Command line
-
hf download hf://spaces/harshitmahour360/ARY/codebase_explanation.md
-
curl -L -o codebase_explanation.md https://huggingface.co/spaces/harshitmahour360/ARY/resolve/main/codebase_explanation.md
SupportDesk Environment Breakdown
This document provides a comprehensive, step-by-step breakdown of how the SupportDesk Environment repository works, what each file does, and how the entire system functions from end-to-end.
1. The High-Level Concept
This repository is an "OpenEnv" style simulation, similar to a flight simulator, but designed for an AI customer support agent.
When evaluating an AI for customer support, it's risky to test it on real unhappy customers immediately. Instead, it's placed in this simulation:
- It simulates angry customers creating tickets.
- It accepts actions taken by an AI agent (e.g., "route to billing" or "send a refund").
- It grades those actions probabilistically based on predefined rules, determining if the AI followed company policy correctly.
2. File-by-File Breakdown
A. Core Configuration
Dockerfile: A blueprint for creating a standardized running environment. It bundles the code and dependencies (like FastAPI) into a container, ensuring the application runs identically on any computer or cloud server..env: Securely stores secret keys likeHF_TOKENand connection URLs, keeping them out of the source code.pyproject.toml: The modern Python configuration file. It lists external packages required by the project (e.g.,fastapi,requests,pydantic).
B. The Rules of the Simulation (supportdesk_env/)
task_bank.py: The central repository for scenarios. It defines three distinct customer issues:- Easy: A customer locked out after a password reset.
- Medium: A customer charged twice requesting a refund.
- Hard: A massive European server outage requiring coordination. It also strictly defines "correct" actions (e.g., "The target team must be billing and you must mention 3-5 business days").
models.py: Defines data structures using Pydantic. It dictates exactly what fields an AI agent can include in an action (team,priority,message,refund_amount) and what information the AI will receive as an observation.grader.py: The evaluation engine. When the AI takes an action, this script compares it against the rules intask_bank.py. It deducts points if the AI hallucinated, used forbidden phrases, or routed the ticket incorrectly, generating a score from 0.0 to 1.0.
C. The Simulation Server (supportdesk_env/server/)
environment.py: Functions as the game engine. It tracks the "state" (current turn, AI's score). When a new task begins, it selects a scenario from thetask_bank. When the AI executes a step, it passes the action to thegrader, records the score, and advances the turn.app.py: The web interface. It utilizes FastAPI to expose the game engine (environment.py) over HTTP, allowing external AI scripts to interact with the simulation via requests (/reset,/step,/health).
D. The AI Player
client.py: A helper utility. Instead of making raw HTTP requests, the AI player script can use this Python object to easily communicate with the server (e.g.,client.reset(),client.step()).inference.py: The primary "Agent Player" testing script. It serves as an example of an AI attempting to navigate the simulation scenarios.
3. Step-by-Step Flow: What Happens When You Run It?
When you execute python inference.py, the following chronological sequence of events occurs simultaneously on the server and the client:
Step 1: Initialization
- In
inference.py, the script loads the.envfile credentials. - It initializes an
OpenAIclient (configured to communicate with Hugging Face). - It initializes the
SupportDeskClient, enabling it to communicate with the locally runninguvicornserver instance ofapp.py.
Step 2: Starting a Scenario (The Task)
- The
inferencescript asks the server to start a scenario usingenv.reset(task_id="login_lockout"). - The server (
environment.py) loads the "Easy" ticket fromtask_bank.py. - The server replies with a
SupportObservationcontaining the simulated customer's email summary.
Step 3: The AI "Thinks"
- In
inference.py, the function_call_model_actionpackages the rules and the customer's email into a comprehensive prompt. - It sends this prompt to the AI Model hosted on Hugging Face using its API key.
- The model generates a JSON object representing a recommended action (e.g.,
{"action_type": "finalize", "team": "account_access", "message": "Try checking your spam..."}).
Step 4: Grading the AI (The Step)
- The client sends the generated JSON action back to the server using
env.step(action). - The server receives it and passes it to
grader.py. grader.pycompares the AI's JSON against the strict expected answers intask_bank.py, checking for proper routing and forbidden terms.- It assigns a score and generates feedback (e.g., "Good progress, but you forgot to set the severity.").
Step 5: Iteration and Result
- The server updates its internal state and returns the score back to the
inferencescript. - The loop terminates if the score is perfect (1.0) or if the AI exhausts its allotted attempts (4 turns).
- The script proceeds to the next task (Medium, then Hard), eventually printing the Mean Score across all simulated tasks.
Summary Statement
This codebase is an automated evaluation environment for customer support AI agents. It consists of a FastAPI server holding three standard tricky scenarios. It evaluates agents based on how well they route tickets, adhere to tone constraints, and follow strict operational policies, grading them instantly using a deterministic rubric. The inference.py script acts as the test runner, plugging a real LLM into the local server to see how well it scores against our constraints.