meta_hackathon / README.md
Neetu143's picture
Upload support triage environment
cfa210f verified
|
Raw History Blame Contribute Delete
9.77 kB
metadata
title: Support Triage Environment
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
pinned: false
tags:
  - openenv
  - fastapi
  - customer-support

Support Triage Environment

support_triage_env is a procedural customer-support operations environment for evaluating agents on evidence-driven triage. The agent receives a real support incident, investigates backend evidence through lookup actions, updates an internal workspace, writes an internal note, drafts a customer-safe reply, and submits a final workflow decision.

Why this environment

This environment models a real operations workflow that human support leads and trust-and-safety analysts already do:

  • investigate backend evidence before promising an outcome
  • choose the correct queue, priority, and status from policy plus operational context
  • capture internal reasoning in an audit-friendly note
  • draft a customer-safe reply that reflects what the backend actually supports

The key design goal is to punish shallow β€œform filling.” An agent can no longer get a high score just by memorizing queue and status values. It must look up order, account, policy, or fraud signals first, and the grader rewards or penalizes that investigation path directly.

The public observation intentionally omits exact grader targets such as per-dimension score breakdowns and literal reply/note requirement slots. The agent sees the task, policies, backend evidence, and workspace state, then has to infer the correct action from that evidence.

Task families

The environment ships with three task families and seeded procedural variants inside each family:

  1. billing_duplicate_charge (easy) Investigate whether a reported double charge is a true duplicate capture or only an authorization hold.
  2. shipping_damaged_replacement (medium) Decide whether a damaged replacement request still needs evidence or is already ready for dispatch.
  3. account_takeover_fraud (hard) Investigate account takeover evidence, linked order state, and fraud telemetry before escalating a security incident.

Default reset(task_id=...) uses the primary variant for reproducible baselines. reset(task_id=..., seed=n) selects alternate variants deterministically.

Grading model

Each case uses a deterministic grader with a 0.0 to 1.0 score built from:

  • investigation coverage
  • priority correctness
  • queue correctness
  • status correctness
  • required tags
  • internal note coverage
  • reply coverage

This is intentionally not a binary β€œsubmit or fail” grader. Agents get partial credit for useful progress and lose ceiling score if they skip evidence gathering. The reported task score is kept strictly inside (0, 1), so fully solved episodes land at 0.99 rather than 1.00.

Action space

The environment accepts a single typed action model:

  • lookup_policy(value)
  • lookup_order(value)
  • lookup_account(value)
  • lookup_fraud_signal(value)
  • set_priority(value)
  • set_queue(value)
  • set_status(value)
  • add_tag(value)
  • remove_tag(value)
  • add_internal_note(value)
  • draft_reply(value)
  • submit()

Allowed values are exposed in every observation.

Observation space

Each observation includes:

  • task_id, difficulty, case_title, and scenario_summary
  • the source ticket and public policy notes
  • investigation requirements and discovered investigation_log records
  • the current structured workspace
  • lookup targets for policy, order, account, and fraud actions
  • valid actions and allowed values
  • last_action_error
  • steps_remaining

Core interface

The repo exposes two environment layers:

  • SupportTriageCoreEnvironment for the strict task API: reset() -> TriageObservation step(action) -> TriageStepResult state() -> TriageState
  • SupportTriageEnvironment as the OpenEnv HTTP adapter used by openenv validate and the HF Space

Typed result models include:

  • InvestigationRecord
  • TriageReward
  • TriageStepInfo
  • TriageStepResult

Reward design

Rewards are incremental:

  • new required lookups raise score immediately
  • correct queue / priority / status updates raise score incrementally
  • internal notes and customer replies earn partial credit when they cover the right facts
  • repeated, invalid, or regressive actions lose reward
  • submitting early loses the completion bonus
  • hitting the step limit ends the episode with a penalty

Project layout

.
β”œβ”€β”€ Dockerfile
β”œβ”€β”€ inference.py
β”œβ”€β”€ openenv.yaml
β”œβ”€β”€ pyproject.toml
β”œβ”€β”€ support_triage_env/environment.py
β”œβ”€β”€ server/
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ app.py
β”‚   └── environment.py
β”œβ”€β”€ support_triage_env/
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ client.py
β”‚   β”œβ”€β”€ models.py
β”‚   β”œβ”€β”€ tasks.py
β”‚   └── server/
β”‚       β”œβ”€β”€ __init__.py
β”‚       β”œβ”€β”€ app.py
β”‚       └── environment.py
└── tests/
    β”œβ”€β”€ test_environment.py
    └── test_inference.py

Local setup

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
pytest

Run the server

source .venv/bin/activate
uvicorn server.app:app --host 0.0.0.0 --port 8000

You can also use the packaged entry point:

source .venv/bin/activate
server

Run with Docker

docker build -t support-triage-env .
docker run --rm -p 8000:8000 support-triage-env

Run the baseline inference script

The baseline uses the OpenAI Python client and accepts either the standard OpenAI variable name or the hackathon-specific one:

  • API_BASE_URL default: https://api.openai.com/v1
  • MODEL_NAME default: gpt-4.1-mini
  • OPENAI_API_KEY or HF_TOKEN required
source .venv/bin/activate
python inference.py

For local runs, inference.py will automatically load variables from .env if that file exists.

Important: The hackathon requires the OpenAI client library, but that does not force you to use OpenAI's own endpoint. The script prefers OPENAI_API_KEY when present, then falls back to HF_TOKEN. Valid setups are:

  1. OpenAI endpoint: API_BASE_URL=https://api.openai.com/v1 OPENAI_API_KEY=<your OpenAI API key> MODEL_NAME=gpt-4.1-mini

  2. Hugging Face router with the OpenAI client: API_BASE_URL=https://router.huggingface.co/v1 HF_TOKEN=hf_... MODEL_NAME=<a Hugging Face router-supported chat model>

  3. Gemini with the OpenAI-compatible endpoint: API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/ OPENAI_API_KEY=<your Gemini API key> or HF_TOKEN=<your Gemini API key> MODEL_NAME=gemini-3-flash-preview

Invalid combination: If HF_TOKEN starts with hf_, do not leave API_BASE_URL set to https://api.openai.com/v1. Your token and API_BASE_URL must belong to the same provider.

The script emits only the required log lines:

  • [START]
  • [STEP]
  • [END]

Baseline performance

Measured locally on April 12, 2026 with a deterministic no-credit fallback run through the actual inference.py entrypoint:

  • OpenAI Python client
  • API_BASE_URL=http://127.0.0.1:9/v1
  • MODEL_NAME=deterministic-fallback
  • OPENAI_API_KEY=test-key
  • REQUEST_TIMEOUT_S=1

Command:

API_BASE_URL=http://127.0.0.1:9/v1 MODEL_NAME=deterministic-fallback OPENAI_API_KEY=test-key REQUEST_TIMEOUT_S=1 .venv/bin/python inference.py

Task results:

  1. billing_duplicate_charge: success=true, steps=11, score=0.99
  2. shipping_damaged_replacement: success=true, steps=10, score=0.99
  3. account_takeover_fraud: success=true, steps=14, score=0.99

Reference transcript:

[START] task=billing_duplicate_charge env=support_triage_env model=deterministic-fallback
[END] success=true steps=11 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.02,0.02,0.02,0.14,0.19,0.30
[START] task=shipping_damaged_replacement env=support_triage_env model=deterministic-fallback
[END] success=true steps=10 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.04,0.04,0.14,0.19,0.30
[START] task=account_takeover_fraud env=support_triage_env model=deterministic-fallback
[END] success=true steps=14 score=0.99 rewards=0.04,0.04,0.04,0.04,0.09,0.11,0.10,0.01,0.01,0.01,0.01,0.14,0.19,0.30

Docker

docker build -t support-triage-env .
docker run -p 8000:8000 support-triage-env

Hugging Face Space

This README already includes Docker Space front matter and the openenv tag expected by the hackathon notes.

Suggested deployment flow:

  1. Create a new Hugging Face Space with SDK set to Docker.
  2. Push this repository as the Space contents.
  3. Add HF_TOKEN as a Space secret before running the baseline.
  4. Confirm the Space is in the Running state and responds on port 8000.

You can automate Space creation, upload, and secret sync with:

source .venv/bin/activate
export HF_HUB_TOKEN=hf_your_huggingface_hub_token
python scripts/deploy_hf_space.py --repo-id <username-or-org>/<space-name>

Notes:

  • HF_HUB_TOKEN is your Hugging Face Hub token used to create/update the Space.
  • .env is used only as the source of runtime values that will be copied into the Space: API_BASE_URL, MODEL_NAME, and HF_TOKEN.
  • The runtime HF_TOKEN secret inside the Space can be a Gemini key, OpenAI key, or any OpenAI-compatible provider key, as long as it matches API_BASE_URL.

Validation checklist

Before submission you should still:

  1. run .venv/bin/python -m pytest -q
  2. run openenv validate .
  3. refresh the baseline numbers in this README if you change the tasks, rubric, provider, or prompt
  4. redeploy the Hugging Face Docker Space and confirm POST /reset returns 200