--- title: Support Triage Environment colorFrom: blue colorTo: green sdk: docker app_port: 8000 pinned: false tags: - openenv - fastapi - customer-support --- # Support Triage Environment `support_triage_env` is a procedural customer-support operations environment for evaluating agents on evidence-driven triage. The agent receives a real support incident, investigates backend evidence through lookup actions, updates an internal workspace, writes an internal note, drafts a customer-safe reply, and submits a final workflow decision. ## Why this environment This environment models a real operations workflow that human support leads and trust-and-safety analysts already do: - investigate backend evidence before promising an outcome - choose the correct queue, priority, and status from policy plus operational context - capture internal reasoning in an audit-friendly note - draft a customer-safe reply that reflects what the backend actually supports The key design goal is to punish shallow “form filling.” An agent can no longer get a high score just by memorizing queue and status values. It must look up order, account, policy, or fraud signals first, and the grader rewards or penalizes that investigation path directly. The public observation intentionally omits exact grader targets such as per-dimension score breakdowns and literal reply/note requirement slots. The agent sees the task, policies, backend evidence, and workspace state, then has to infer the correct action from that evidence. ## Task families The environment ships with three task families and seeded procedural variants inside each family: 1. `billing_duplicate_charge` (`easy`) Investigate whether a reported double charge is a true duplicate capture or only an authorization hold. 2. `shipping_damaged_replacement` (`medium`) Decide whether a damaged replacement request still needs evidence or is already ready for dispatch. 3. `account_takeover_fraud` (`hard`) Investigate account takeover evidence, linked order state, and fraud telemetry before escalating a security incident. Default `reset(task_id=...)` uses the primary variant for reproducible baselines. `reset(task_id=..., seed=n)` selects alternate variants deterministically. ## Grading model Each case uses a deterministic grader with a `0.0` to `1.0` score built from: - investigation coverage - priority correctness - queue correctness - status correctness - required tags - internal note coverage - reply coverage This is intentionally not a binary “submit or fail” grader. Agents get partial credit for useful progress and lose ceiling score if they skip evidence gathering. The reported task score is kept strictly inside `(0, 1)`, so fully solved episodes land at `0.99` rather than `1.00`. ## Action space The environment accepts a single typed action model: - `lookup_policy(value)` - `lookup_order(value)` - `lookup_account(value)` - `lookup_fraud_signal(value)` - `set_priority(value)` - `set_queue(value)` - `set_status(value)` - `add_tag(value)` - `remove_tag(value)` - `add_internal_note(value)` - `draft_reply(value)` - `submit()` Allowed values are exposed in every observation. ## Observation space Each observation includes: - `task_id`, `difficulty`, `case_title`, and `scenario_summary` - the source ticket and public policy notes - investigation requirements and discovered `investigation_log` records - the current structured workspace - lookup targets for policy, order, account, and fraud actions - valid actions and allowed values - `last_action_error` - `steps_remaining` ## Core interface The repo exposes two environment layers: - `SupportTriageCoreEnvironment` for the strict task API: `reset() -> TriageObservation` `step(action) -> TriageStepResult` `state() -> TriageState` - `SupportTriageEnvironment` as the OpenEnv HTTP adapter used by `openenv validate` and the HF Space Typed result models include: - `InvestigationRecord` - `TriageReward` - `TriageStepInfo` - `TriageStepResult` ## Reward design Rewards are incremental: - new required lookups raise score immediately - correct queue / priority / status updates raise score incrementally - internal notes and customer replies earn partial credit when they cover the right facts - repeated, invalid, or regressive actions lose reward - submitting early loses the completion bonus - hitting the step limit ends the episode with a penalty ## Project layout ```text . ├── Dockerfile ├── inference.py ├── openenv.yaml ├── pyproject.toml ├── support_triage_env/environment.py ├── server/ │ ├── __init__.py │ ├── app.py │ └── environment.py ├── support_triage_env/ │ ├── __init__.py │ ├── client.py │ ├── models.py │ ├── tasks.py │ └── server/ │ ├── __init__.py │ ├── app.py │ └── environment.py └── tests/ ├── test_environment.py └── test_inference.py ``` ## Local setup ```bash python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -e ".[dev]" pytest ``` ## Run the server ```bash source .venv/bin/activate uvicorn server.app:app --host 0.0.0.0 --port 8000 ``` You can also use the packaged entry point: ```bash source .venv/bin/activate server ``` ## Run with Docker ```bash docker build -t support-triage-env . docker run --rm -p 8000:8000 support-triage-env ``` ## Run the baseline inference script The baseline uses the OpenAI Python client and accepts either the standard OpenAI variable name or the hackathon-specific one: - `API_BASE_URL` default: `https://api.openai.com/v1` - `MODEL_NAME` default: `gpt-4.1-mini` - `OPENAI_API_KEY` or `HF_TOKEN` required ```bash source .venv/bin/activate python inference.py ``` For local runs, `inference.py` will automatically load variables from `.env` if that file exists. Important: The hackathon requires the OpenAI client library, but that does not force you to use OpenAI's own endpoint. The script prefers `OPENAI_API_KEY` when present, then falls back to `HF_TOKEN`. Valid setups are: 1. OpenAI endpoint: `API_BASE_URL=https://api.openai.com/v1` `OPENAI_API_KEY=` `MODEL_NAME=gpt-4.1-mini` 2. Hugging Face router with the OpenAI client: `API_BASE_URL=https://router.huggingface.co/v1` `HF_TOKEN=hf_...` `MODEL_NAME=` 3. Gemini with the OpenAI-compatible endpoint: `API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/` `OPENAI_API_KEY=` or `HF_TOKEN=` `MODEL_NAME=gemini-3-flash-preview` Invalid combination: If `HF_TOKEN` starts with `hf_`, do not leave `API_BASE_URL` set to `https://api.openai.com/v1`. Your token and `API_BASE_URL` must belong to the same provider. The script emits only the required log lines: - `[START]` - `[STEP]` - `[END]` ## Baseline performance Measured locally on April 12, 2026 with a deterministic no-credit fallback run through the actual `inference.py` entrypoint: - OpenAI Python client - `API_BASE_URL=http://127.0.0.1:9/v1` - `MODEL_NAME=deterministic-fallback` - `OPENAI_API_KEY=test-key` - `REQUEST_TIMEOUT_S=1` Command: ```bash API_BASE_URL=http://127.0.0.1:9/v1 MODEL_NAME=deterministic-fallback OPENAI_API_KEY=test-key REQUEST_TIMEOUT_S=1 .venv/bin/python inference.py ``` Task results: 1. `billing_duplicate_charge`: `success=true`, `steps=11`, `score=0.99` 2. `shipping_damaged_replacement`: `success=true`, `steps=10`, `score=0.99` 3. `account_takeover_fraud`: `success=true`, `steps=14`, `score=0.99` Reference transcript: ```text [START] task=billing_duplicate_charge env=support_triage_env model=deterministic-fallback [END] success=true steps=11 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.02,0.02,0.02,0.14,0.19,0.30 [START] task=shipping_damaged_replacement env=support_triage_env model=deterministic-fallback [END] success=true steps=10 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.04,0.04,0.14,0.19,0.30 [START] task=account_takeover_fraud env=support_triage_env model=deterministic-fallback [END] success=true steps=14 score=0.99 rewards=0.04,0.04,0.04,0.04,0.09,0.11,0.10,0.01,0.01,0.01,0.01,0.14,0.19,0.30 ``` ## Docker ```bash docker build -t support-triage-env . docker run -p 8000:8000 support-triage-env ``` ## Hugging Face Space This README already includes Docker Space front matter and the `openenv` tag expected by the hackathon notes. Suggested deployment flow: 1. Create a new Hugging Face Space with SDK set to `Docker`. 2. Push this repository as the Space contents. 3. Add `HF_TOKEN` as a Space secret before running the baseline. 4. Confirm the Space is in the `Running` state and responds on port `8000`. You can automate Space creation, upload, and secret sync with: ```bash source .venv/bin/activate export HF_HUB_TOKEN=hf_your_huggingface_hub_token python scripts/deploy_hf_space.py --repo-id / ``` Notes: - `HF_HUB_TOKEN` is your Hugging Face Hub token used to create/update the Space. - `.env` is used only as the source of runtime values that will be copied into the Space: `API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN`. - The runtime `HF_TOKEN` secret inside the Space can be a Gemini key, OpenAI key, or any OpenAI-compatible provider key, as long as it matches `API_BASE_URL`. ## Validation checklist Before submission you should still: 1. run `.venv/bin/python -m pytest -q` 2. run `openenv validate .` 3. refresh the baseline numbers in this README if you change the tasks, rubric, provider, or prompt 4. redeploy the Hugging Face Docker Space and confirm `POST /reset` returns `200`