Spaces:
Sleeping
Sleeping
|
Download README.md from Neetu143/meta_hackathon: direct link, hf CLI and curl.
- Browser
- Download file 9.77 kB
-
https://huggingface.co/spaces/Neetu143/meta_hackathon/resolve/main/README.md
- Command line
-
hf download hf://spaces/Neetu143/meta_hackathon/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Neetu143/meta_hackathon/resolve/main/README.md
9.77 kB
| title: Support Triage Environment | |
| colorFrom: blue | |
| colorTo: green | |
| sdk: docker | |
| app_port: 8000 | |
| pinned: false | |
| tags: | |
| - openenv | |
| - fastapi | |
| - customer-support | |
| # Support Triage Environment | |
| `support_triage_env` is a procedural customer-support operations environment for evaluating agents on evidence-driven triage. The agent receives a real support incident, investigates backend evidence through lookup actions, updates an internal workspace, writes an internal note, drafts a customer-safe reply, and submits a final workflow decision. | |
| ## Why this environment | |
| This environment models a real operations workflow that human support leads and trust-and-safety analysts already do: | |
| - investigate backend evidence before promising an outcome | |
| - choose the correct queue, priority, and status from policy plus operational context | |
| - capture internal reasoning in an audit-friendly note | |
| - draft a customer-safe reply that reflects what the backend actually supports | |
| The key design goal is to punish shallow βform filling.β An agent can no longer get a high score just by memorizing queue and status values. It must look up order, account, policy, or fraud signals first, and the grader rewards or penalizes that investigation path directly. | |
| The public observation intentionally omits exact grader targets such as per-dimension score breakdowns and literal reply/note requirement slots. The agent sees the task, policies, backend evidence, and workspace state, then has to infer the correct action from that evidence. | |
| ## Task families | |
| The environment ships with three task families and seeded procedural variants inside each family: | |
| 1. `billing_duplicate_charge` (`easy`) | |
| Investigate whether a reported double charge is a true duplicate capture or only an authorization hold. | |
| 2. `shipping_damaged_replacement` (`medium`) | |
| Decide whether a damaged replacement request still needs evidence or is already ready for dispatch. | |
| 3. `account_takeover_fraud` (`hard`) | |
| Investigate account takeover evidence, linked order state, and fraud telemetry before escalating a security incident. | |
| Default `reset(task_id=...)` uses the primary variant for reproducible baselines. `reset(task_id=..., seed=n)` selects alternate variants deterministically. | |
| ## Grading model | |
| Each case uses a deterministic grader with a `0.0` to `1.0` score built from: | |
| - investigation coverage | |
| - priority correctness | |
| - queue correctness | |
| - status correctness | |
| - required tags | |
| - internal note coverage | |
| - reply coverage | |
| This is intentionally not a binary βsubmit or failβ grader. Agents get partial credit for useful progress and lose ceiling score if they skip evidence gathering. | |
| The reported task score is kept strictly inside `(0, 1)`, so fully solved episodes land at `0.99` rather than `1.00`. | |
| ## Action space | |
| The environment accepts a single typed action model: | |
| - `lookup_policy(value)` | |
| - `lookup_order(value)` | |
| - `lookup_account(value)` | |
| - `lookup_fraud_signal(value)` | |
| - `set_priority(value)` | |
| - `set_queue(value)` | |
| - `set_status(value)` | |
| - `add_tag(value)` | |
| - `remove_tag(value)` | |
| - `add_internal_note(value)` | |
| - `draft_reply(value)` | |
| - `submit()` | |
| Allowed values are exposed in every observation. | |
| ## Observation space | |
| Each observation includes: | |
| - `task_id`, `difficulty`, `case_title`, and `scenario_summary` | |
| - the source ticket and public policy notes | |
| - investigation requirements and discovered `investigation_log` records | |
| - the current structured workspace | |
| - lookup targets for policy, order, account, and fraud actions | |
| - valid actions and allowed values | |
| - `last_action_error` | |
| - `steps_remaining` | |
| ## Core interface | |
| The repo exposes two environment layers: | |
| - `SupportTriageCoreEnvironment` for the strict task API: | |
| `reset() -> TriageObservation` | |
| `step(action) -> TriageStepResult` | |
| `state() -> TriageState` | |
| - `SupportTriageEnvironment` as the OpenEnv HTTP adapter used by `openenv validate` and the HF Space | |
| Typed result models include: | |
| - `InvestigationRecord` | |
| - `TriageReward` | |
| - `TriageStepInfo` | |
| - `TriageStepResult` | |
| ## Reward design | |
| Rewards are incremental: | |
| - new required lookups raise score immediately | |
| - correct queue / priority / status updates raise score incrementally | |
| - internal notes and customer replies earn partial credit when they cover the right facts | |
| - repeated, invalid, or regressive actions lose reward | |
| - submitting early loses the completion bonus | |
| - hitting the step limit ends the episode with a penalty | |
| ## Project layout | |
| ```text | |
| . | |
| βββ Dockerfile | |
| βββ inference.py | |
| βββ openenv.yaml | |
| βββ pyproject.toml | |
| βββ support_triage_env/environment.py | |
| βββ server/ | |
| β βββ __init__.py | |
| β βββ app.py | |
| β βββ environment.py | |
| βββ support_triage_env/ | |
| β βββ __init__.py | |
| β βββ client.py | |
| β βββ models.py | |
| β βββ tasks.py | |
| β βββ server/ | |
| β βββ __init__.py | |
| β βββ app.py | |
| β βββ environment.py | |
| βββ tests/ | |
| βββ test_environment.py | |
| βββ test_inference.py | |
| ``` | |
| ## Local setup | |
| ```bash | |
| python3 -m venv .venv | |
| source .venv/bin/activate | |
| python -m pip install --upgrade pip | |
| python -m pip install -e ".[dev]" | |
| pytest | |
| ``` | |
| ## Run the server | |
| ```bash | |
| source .venv/bin/activate | |
| uvicorn server.app:app --host 0.0.0.0 --port 8000 | |
| ``` | |
| You can also use the packaged entry point: | |
| ```bash | |
| source .venv/bin/activate | |
| server | |
| ``` | |
| ## Run with Docker | |
| ```bash | |
| docker build -t support-triage-env . | |
| docker run --rm -p 8000:8000 support-triage-env | |
| ``` | |
| ## Run the baseline inference script | |
| The baseline uses the OpenAI Python client and accepts either the standard OpenAI variable name or the hackathon-specific one: | |
| - `API_BASE_URL` default: `https://api.openai.com/v1` | |
| - `MODEL_NAME` default: `gpt-4.1-mini` | |
| - `OPENAI_API_KEY` or `HF_TOKEN` required | |
| ```bash | |
| source .venv/bin/activate | |
| python inference.py | |
| ``` | |
| For local runs, `inference.py` will automatically load variables from `.env` if that file exists. | |
| Important: | |
| The hackathon requires the OpenAI client library, but that does not force you to use OpenAI's | |
| own endpoint. The script prefers `OPENAI_API_KEY` when present, then falls back to `HF_TOKEN`. | |
| Valid setups are: | |
| 1. OpenAI endpoint: | |
| `API_BASE_URL=https://api.openai.com/v1` | |
| `OPENAI_API_KEY=<your OpenAI API key>` | |
| `MODEL_NAME=gpt-4.1-mini` | |
| 2. Hugging Face router with the OpenAI client: | |
| `API_BASE_URL=https://router.huggingface.co/v1` | |
| `HF_TOKEN=hf_...` | |
| `MODEL_NAME=<a Hugging Face router-supported chat model>` | |
| 3. Gemini with the OpenAI-compatible endpoint: | |
| `API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/` | |
| `OPENAI_API_KEY=<your Gemini API key>` or `HF_TOKEN=<your Gemini API key>` | |
| `MODEL_NAME=gemini-3-flash-preview` | |
| Invalid combination: | |
| If `HF_TOKEN` starts with `hf_`, do not leave `API_BASE_URL` set to | |
| `https://api.openai.com/v1`. Your token and `API_BASE_URL` must belong to the same provider. | |
| The script emits only the required log lines: | |
| - `[START]` | |
| - `[STEP]` | |
| - `[END]` | |
| ## Baseline performance | |
| Measured locally on April 12, 2026 with a deterministic no-credit fallback run through the actual `inference.py` entrypoint: | |
| - OpenAI Python client | |
| - `API_BASE_URL=http://127.0.0.1:9/v1` | |
| - `MODEL_NAME=deterministic-fallback` | |
| - `OPENAI_API_KEY=test-key` | |
| - `REQUEST_TIMEOUT_S=1` | |
| Command: | |
| ```bash | |
| API_BASE_URL=http://127.0.0.1:9/v1 MODEL_NAME=deterministic-fallback OPENAI_API_KEY=test-key REQUEST_TIMEOUT_S=1 .venv/bin/python inference.py | |
| ``` | |
| Task results: | |
| 1. `billing_duplicate_charge`: `success=true`, `steps=11`, `score=0.99` | |
| 2. `shipping_damaged_replacement`: `success=true`, `steps=10`, `score=0.99` | |
| 3. `account_takeover_fraud`: `success=true`, `steps=14`, `score=0.99` | |
| Reference transcript: | |
| ```text | |
| [START] task=billing_duplicate_charge env=support_triage_env model=deterministic-fallback | |
| [END] success=true steps=11 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.02,0.02,0.02,0.14,0.19,0.30 | |
| [START] task=shipping_damaged_replacement env=support_triage_env model=deterministic-fallback | |
| [END] success=true steps=10 score=0.99 rewards=0.10,0.10,0.09,0.11,0.10,0.04,0.04,0.14,0.19,0.30 | |
| [START] task=account_takeover_fraud env=support_triage_env model=deterministic-fallback | |
| [END] success=true steps=14 score=0.99 rewards=0.04,0.04,0.04,0.04,0.09,0.11,0.10,0.01,0.01,0.01,0.01,0.14,0.19,0.30 | |
| ``` | |
| ## Docker | |
| ```bash | |
| docker build -t support-triage-env . | |
| docker run -p 8000:8000 support-triage-env | |
| ``` | |
| ## Hugging Face Space | |
| This README already includes Docker Space front matter and the `openenv` tag expected by the hackathon notes. | |
| Suggested deployment flow: | |
| 1. Create a new Hugging Face Space with SDK set to `Docker`. | |
| 2. Push this repository as the Space contents. | |
| 3. Add `HF_TOKEN` as a Space secret before running the baseline. | |
| 4. Confirm the Space is in the `Running` state and responds on port `8000`. | |
| You can automate Space creation, upload, and secret sync with: | |
| ```bash | |
| source .venv/bin/activate | |
| export HF_HUB_TOKEN=hf_your_huggingface_hub_token | |
| python scripts/deploy_hf_space.py --repo-id <username-or-org>/<space-name> | |
| ``` | |
| Notes: | |
| - `HF_HUB_TOKEN` is your Hugging Face Hub token used to create/update the Space. | |
| - `.env` is used only as the source of runtime values that will be copied into the Space: | |
| `API_BASE_URL`, `MODEL_NAME`, and `HF_TOKEN`. | |
| - The runtime `HF_TOKEN` secret inside the Space can be a Gemini key, OpenAI key, or any | |
| OpenAI-compatible provider key, as long as it matches `API_BASE_URL`. | |
| ## Validation checklist | |
| Before submission you should still: | |
| 1. run `.venv/bin/python -m pytest -q` | |
| 2. run `openenv validate .` | |
| 3. refresh the baseline numbers in this README if you change the tasks, rubric, provider, or prompt | |
| 4. redeploy the Hugging Face Docker Space and confirm `POST /reset` returns `200` | |