Spaces:
Sleeping
Download README.md from Neetu143/meta_hackathon: direct link, hf CLI and curl.
- Browser
- Download file 9.77 kB
-
https://huggingface.co/spaces/Neetu143/meta_hackathon/resolve/main/README.md
- Command line
-
hf download hf://spaces/Neetu143/meta_hackathon/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Neetu143/meta_hackathon/resolve/main/README.md
title: Support Triage Environment
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
pinned: false
tags:
- openenv
- fastapi
- customer-support
Support Triage Environment
support_triage_env is a procedural customer-support operations environment for evaluating agents on evidence-driven triage. The agent receives a real support incident, investigates backend evidence through lookup actions, updates an internal workspace, writes an internal note, drafts a customer-safe reply, and submits a final workflow decision.
Why this environment
This environment models a real operations workflow that human support leads and trust-and-safety analysts already do:
- investigate backend evidence before promising an outcome
- choose the correct queue, priority, and status from policy plus operational context
- capture internal reasoning in an audit-friendly note
- draft a customer-safe reply that reflects what the backend actually supports
The key design goal is to punish shallow βform filling.β An agent can no longer get a high score just by memorizing queue and status values. It must look up order, account, policy, or fraud signals first, and the grader rewards or penalizes that investigation path directly.
The public observation intentionally omits exact grader targets such as per-dimension score breakdowns and literal reply/note requirement slots. The agent sees the task, policies, backend evidence, and workspace state, then has to infer the correct action from that evidence.
Task families
The environment ships with three task families and seeded procedural variants inside each family:
billing_duplicate_charge(easy) Investigate whether a reported double charge is a true duplicate capture or only an authorization hold.shipping_damaged_replacement(medium) Decide whether a damaged replacement request still needs evidence or is already ready for dispatch.account_takeover_fraud(hard) Investigate account takeover evidence, linked order state, and fraud telemetry before escalating a security incident.
Default reset(task_id=...) uses the primary variant for reproducible baselines. reset(task_id=..., seed=n) selects alternate variants deterministically.
Grading model
Each case uses a deterministic grader with a 0.0 to 1.0 score built from:
- investigation coverage
- priority correctness
- queue correctness
- status correctness
- required tags
- internal note coverage
- reply coverage
This is intentionally not a binary βsubmit or failβ grader. Agents get partial credit for useful progress and lose ceiling score if they skip evidence gathering.
The reported task score is kept strictly inside (0, 1), so fully solved episodes land at 0.99 rather than 1.00.
Action space
The environment accepts a single typed action model:
lookup_policy(value)lookup_order(value)lookup_account(value)lookup_fraud_signal(value)set_priority(value)set_queue(value)set_status(value)add_tag(value)remove_tag(value)add_internal_note(value)draft_reply(value)submit()
Allowed values are exposed in every observation.
Observation space
Each observation includes:
task_id,difficulty,case_title, andscenario_summary- the source ticket and public policy notes
- investigation requirements and discovered
investigation_logrecords - the current structured workspace
- lookup targets for policy, order, account, and fraud actions
- valid actions and allowed values
last_action_errorsteps_remaining
Core interface
The repo exposes two environment layers:
SupportTriageCoreEnvironmentfor the strict task API:reset() -> TriageObservationstep(action) -> TriageStepResultstate() -> TriageStateSupportTriageEnvironmentas the OpenEnv HTTP adapter used byopenenv validateand the HF Space
Typed result models include:
InvestigationRecordTriageRewardTriageStepInfoTriageStepResult
Reward design
Rewards are incremental:
- new required lookups raise score immediately
- correct queue / priority / status updates raise score incrementally
- internal notes and customer replies earn partial credit when they cover the right facts
- repeated, invalid, or regressive actions lose reward
- submitting early loses the completion bonus
- hitting the step limit ends the episode with a penalty
Project layout
.
βββ Dockerfile
βββ inference.py
βββ openenv.yaml
βββ pyproject.toml
βββ support_triage_env/environment.py
βββ server/
β βββ __init__.py
β βββ app.py
β βββ environment.py
βββ support_triage_env/
β βββ __init__.py
β βββ client.py
β βββ models.py
β βββ tasks.py
β βββ server/
β βββ __init__.py
β βββ app.py
β βββ environment.py
βββ tests/
βββ test_environment.py
βββ test_inference.py
Local setup
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
pytest
Run the server
source .venv/bin/activate
uvicorn server.app:app --host 0.0.0.0 --port 8000
You can also use the packaged entry point:
source .venv/bin/activate
server
Run with Docker
docker build -t support-triage-env .
docker run --rm -p 8000:8000 support-triage-env
Run the baseline inference script
The baseline uses the OpenAI Python client and accepts either the standard OpenAI variable name or the hackathon-specific one:
API_BASE_URLdefault:https://api.openai.com/v1MODEL_NAMEdefault:gpt-4.1-miniOPENAI_API_KEYorHF_TOKENrequired
source .venv/bin/activate
python inference.py
For local runs, inference.py will automatically load variables from .env if that file exists.
Important:
The hackathon requires the OpenAI client library, but that does not force you to use OpenAI's
own endpoint. The script prefers OPENAI_API_KEY when present, then falls back to HF_TOKEN.
Valid setups are:
OpenAI endpoint:
API_BASE_URL=https://api.openai.com/v1OPENAI_API_KEY=<your OpenAI API key>MODEL_NAME=gpt-4.1-miniHugging Face router with the OpenAI client:
API_BASE_URL=https://router.huggingface.co/v1HF_TOKEN=hf_...MODEL_NAME=<a Hugging Face router-supported chat model>Gemini with the OpenAI-compatible endpoint:
API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/OPENAI_API_KEY=<your Gemini API key>orHF_TOKEN=<your Gemini API key>MODEL_NAME=gemini-3-flash-preview
Invalid combination:
If HF_TOKEN starts with hf_, do not leave API_BASE_URL set to
https://api.openai.com/v1. Your token and API_BASE_URL must belong to the same provider.
The script emits only the required log lines:
[START][STEP][END]
Baseline performance
Measured locally on April 12, 2026 with a deterministic no-credit fallback run through the actual inference.py entrypoint:
- OpenAI Python client
API_BASE_URL=http://127.0.0.1:9/v1MODEL_NAME=deterministic-fallbackOPENAI_API_KEY=test-keyREQUEST_TIMEOUT_S=1
Command:
API_BASE_URL=http://127.0.0.1:9/v1 MODEL_NAME=deterministic-fallback OPENAI_API_KEY=test-key REQUEST_TIMEOUT_S=1 .venv/bin/python inference.py
Task results:
billing_duplicate_charge:success=true,steps=11,score=0.99shipping_damaged_replacement:success=true,steps=10,score=0.99account_takeover_fraud:success=true,steps=14,score=0.99
Reference transcript:
[START] task=billing_duplicate_charge env=support_triage_env model=deterministic-fallback
[END] success=true steps=11 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.02,0.02,0.02,0.14,0.19,0.30
[START] task=shipping_damaged_replacement env=support_triage_env model=deterministic-fallback
[END] success=true steps=10 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.04,0.04,0.14,0.19,0.30
[START] task=account_takeover_fraud env=support_triage_env model=deterministic-fallback
[END] success=true steps=14 score=0.99 rewards=0.04,0.04,0.04,0.04,0.09,0.11,0.10,0.01,0.01,0.01,0.01,0.14,0.19,0.30
Docker
docker build -t support-triage-env .
docker run -p 8000:8000 support-triage-env
Hugging Face Space
This README already includes Docker Space front matter and the openenv tag expected by the hackathon notes.
Suggested deployment flow:
- Create a new Hugging Face Space with SDK set to
Docker. - Push this repository as the Space contents.
- Add
HF_TOKENas a Space secret before running the baseline. - Confirm the Space is in the
Runningstate and responds on port8000.
You can automate Space creation, upload, and secret sync with:
source .venv/bin/activate
export HF_HUB_TOKEN=hf_your_huggingface_hub_token
python scripts/deploy_hf_space.py --repo-id <username-or-org>/<space-name>
Notes:
HF_HUB_TOKENis your Hugging Face Hub token used to create/update the Space..envis used only as the source of runtime values that will be copied into the Space:API_BASE_URL,MODEL_NAME, andHF_TOKEN.- The runtime
HF_TOKENsecret inside the Space can be a Gemini key, OpenAI key, or any OpenAI-compatible provider key, as long as it matchesAPI_BASE_URL.
Validation checklist
Before submission you should still:
- run
.venv/bin/python -m pytest -q - run
openenv validate . - refresh the baseline numbers in this README if you change the tasks, rubric, provider, or prompt
- redeploy the Hugging Face Docker Space and confirm
POST /resetreturns200