meta_hackathon / README.md
Neetu143's picture
Upload support triage environment
cfa210f verified
|
Raw History Blame Contribute Delete
9.77 kB
---
title: Support Triage Environment
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
pinned: false
tags:
- openenv
- fastapi
- customer-support
---
# Support Triage Environment
`support_triage_env` is a procedural customer-support operations environment for evaluating agents on evidence-driven triage. The agent receives a real support incident, investigates backend evidence through lookup actions, updates an internal workspace, writes an internal note, drafts a customer-safe reply, and submits a final workflow decision.
## Why this environment
This environment models a real operations workflow that human support leads and trust-and-safety analysts already do:
- investigate backend evidence before promising an outcome
- choose the correct queue, priority, and status from policy plus operational context
- capture internal reasoning in an audit-friendly note
- draft a customer-safe reply that reflects what the backend actually supports
The key design goal is to punish shallow β€œform filling.” An agent can no longer get a high score just by memorizing queue and status values. It must look up order, account, policy, or fraud signals first, and the grader rewards or penalizes that investigation path directly.
The public observation intentionally omits exact grader targets such as per-dimension score breakdowns and literal reply/note requirement slots. The agent sees the task, policies, backend evidence, and workspace state, then has to infer the correct action from that evidence.
## Task families
The environment ships with three task families and seeded procedural variants inside each family:
1. `billing_duplicate_charge` (`easy`)
Investigate whether a reported double charge is a true duplicate capture or only an authorization hold.
2. `shipping_damaged_replacement` (`medium`)
Decide whether a damaged replacement request still needs evidence or is already ready for dispatch.
3. `account_takeover_fraud` (`hard`)
Investigate account takeover evidence, linked order state, and fraud telemetry before escalating a security incident.
Default `reset(task_id=...)` uses the primary variant for reproducible baselines. `reset(task_id=..., seed=n)` selects alternate variants deterministically.
## Grading model
Each case uses a deterministic grader with a `0.0` to `1.0` score built from:
- investigation coverage
- priority correctness
- queue correctness
- status correctness
- required tags
- internal note coverage
- reply coverage
This is intentionally not a binary β€œsubmit or fail” grader. Agents get partial credit for useful progress and lose ceiling score if they skip evidence gathering.
The reported task score is kept strictly inside `(0, 1)`, so fully solved episodes land at `0.99` rather than `1.00`.
## Action space
The environment accepts a single typed action model:
- `lookup_policy(value)`
- `lookup_order(value)`
- `lookup_account(value)`
- `lookup_fraud_signal(value)`
- `set_priority(value)`
- `set_queue(value)`
- `set_status(value)`
- `add_tag(value)`
- `remove_tag(value)`
- `add_internal_note(value)`
- `draft_reply(value)`
- `submit()`
Allowed values are exposed in every observation.
## Observation space
Each observation includes:
- `task_id`, `difficulty`, `case_title`, and `scenario_summary`
- the source ticket and public policy notes
- investigation requirements and discovered `investigation_log` records
- the current structured workspace
- lookup targets for policy, order, account, and fraud actions
- valid actions and allowed values
- `last_action_error`
- `steps_remaining`
## Core interface
The repo exposes two environment layers:
- `SupportTriageCoreEnvironment` for the strict task API:
`reset() -> TriageObservation`
`step(action) -> TriageStepResult`
`state() -> TriageState`
- `SupportTriageEnvironment` as the OpenEnv HTTP adapter used by `openenv validate` and the HF Space
Typed result models include:
- `InvestigationRecord`
- `TriageReward`
- `TriageStepInfo`
- `TriageStepResult`
## Reward design
Rewards are incremental:
- new required lookups raise score immediately
- correct queue / priority / status updates raise score incrementally
- internal notes and customer replies earn partial credit when they cover the right facts
- repeated, invalid, or regressive actions lose reward
- submitting early loses the completion bonus
- hitting the step limit ends the episode with a penalty
## Project layout
```text
.
β”œβ”€β”€ Dockerfile
β”œβ”€β”€ inference.py
β”œβ”€β”€ openenv.yaml
β”œβ”€β”€ pyproject.toml
β”œβ”€β”€ support_triage_env/environment.py
β”œβ”€β”€ server/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ app.py
β”‚ └── environment.py
β”œβ”€β”€ support_triage_env/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ client.py
β”‚ β”œβ”€β”€ models.py
β”‚ β”œβ”€β”€ tasks.py
β”‚ └── server/
β”‚ β”œβ”€β”€ __init__.py
β”‚ β”œβ”€β”€ app.py
β”‚ └── environment.py
└── tests/
β”œβ”€β”€ test_environment.py
└── test_inference.py
```
## Local setup
```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
pytest
```
## Run the server
```bash
source .venv/bin/activate
uvicorn server.app:app --host 0.0.0.0 --port 8000
```
You can also use the packaged entry point:
```bash
source .venv/bin/activate
server
```
## Run with Docker
```bash
docker build -t support-triage-env .
docker run --rm -p 8000:8000 support-triage-env
```
## Run the baseline inference script
The baseline uses the OpenAI Python client and accepts either the standard OpenAI variable name or the hackathon-specific one:
- `API_BASE_URL` default: `https://api.openai.com/v1`
- `MODEL_NAME` default: `gpt-4.1-mini`
- `OPENAI_API_KEY` or `HF_TOKEN` required
```bash
source .venv/bin/activate
python inference.py
```
For local runs, `inference.py` will automatically load variables from `.env` if that file exists.
Important:
The hackathon requires the OpenAI client library, but that does not force you to use OpenAI's
own endpoint. The script prefers `OPENAI_API_KEY` when present, then falls back to `HF_TOKEN`.
Valid setups are:
1. OpenAI endpoint:
`API_BASE_URL=https://api.openai.com/v1`
`OPENAI_API_KEY=<your OpenAI API key>`
`MODEL_NAME=gpt-4.1-mini`
2. Hugging Face router with the OpenAI client:
`API_BASE_URL=https://router.huggingface.co/v1`
`HF_TOKEN=hf_...`
`MODEL_NAME=<a Hugging Face router-supported chat model>`
3. Gemini with the OpenAI-compatible endpoint:
`API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/`
`OPENAI_API_KEY=<your Gemini API key>` or `HF_TOKEN=<your Gemini API key>`
`MODEL_NAME=gemini-3-flash-preview`
Invalid combination:
If `HF_TOKEN` starts with `hf_`, do not leave `API_BASE_URL` set to
`https://api.openai.com/v1`. Your token and `API_BASE_URL` must belong to the same provider.
The script emits only the required log lines:
- `[START]`
- `[STEP]`
- `[END]`
## Baseline performance
Measured locally on April 12, 2026 with a deterministic no-credit fallback run through the actual `inference.py` entrypoint:
- OpenAI Python client
- `API_BASE_URL=http://127.0.0.1:9/v1`
- `MODEL_NAME=deterministic-fallback`
- `OPENAI_API_KEY=test-key`
- `REQUEST_TIMEOUT_S=1`
Command:
```bash
API_BASE_URL=http://127.0.0.1:9/v1 MODEL_NAME=deterministic-fallback OPENAI_API_KEY=test-key REQUEST_TIMEOUT_S=1 .venv/bin/python inference.py
```
Task results:
1. `billing_duplicate_charge`: `success=true`, `steps=11`, `score=0.99`
2. `shipping_damaged_replacement`: `success=true`, `steps=10`, `score=0.99`
3. `account_takeover_fraud`: `success=true`, `steps=14`, `score=0.99`
Reference transcript:
```text
[START] task=billing_duplicate_charge env=support_triage_env model=deterministic-fallback
[END] success=true steps=11 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.02,0.02,0.02,0.14,0.19,0.30
[START] task=shipping_damaged_replacement env=support_triage_env model=deterministic-fallback
[END] success=true steps=10 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.04,0.04,0.14,0.19,0.30
[START] task=account_takeover_fraud env=support_triage_env model=deterministic-fallback
[END] success=true steps=14 score=0.99 rewards=0.04,0.04,0.04,0.04,0.09,0.11,0.10,0.01,0.01,0.01,0.01,0.14,0.19,0.30
```
## Docker
```bash
docker build -t support-triage-env .
docker run -p 8000:8000 support-triage-env
```
## Hugging Face Space
This README already includes Docker Space front matter and the `openenv` tag expected by the hackathon notes.
Suggested deployment flow:
1. Create a new Hugging Face Space with SDK set to `Docker`.
2. Push this repository as the Space contents.
3. Add `HF_TOKEN` as a Space secret before running the baseline.
4. Confirm the Space is in the `Running` state and responds on port `8000`.
You can automate Space creation, upload, and secret sync with:
```bash
source .venv/bin/activate
export HF_HUB_TOKEN=hf_your_huggingface_hub_token
python scripts/deploy_hf_space.py --repo-id <username-or-org>/<space-name>
```
Notes:
- `HF_HUB_TOKEN` is your Hugging Face Hub token used to create/update the Space.
- `.env` is used only as the source of runtime values that will be copied into the Space:
`API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN`.
- The runtime `HF_TOKEN` secret inside the Space can be a Gemini key, OpenAI key, or any
OpenAI-compatible provider key, as long as it matches `API_BASE_URL`.
## Validation checklist
Before submission you should still:
1. run `.venv/bin/python -m pytest -q`
2. run `openenv validate .`
3. refresh the baseline numbers in this README if you change the tasks, rubric, provider, or prompt
4. redeploy the Hugging Face Docker Space and confirm `POST /reset` returns `200`