| --- |
| title: PolicyEvolverEnv |
| colorFrom: blue |
| colorTo: indigo |
| sdk: docker |
| app_port: 8000 |
| base_path: /dashboard/ |
| --- |
| |
| # PolicyEvolverEnv |
|
|
| ## 1. Environment Overview and Motivation |
|
|
| **PolicyEvolverEnv** is an OpenEnv-compliant reinforcement learning environment where an AI agent learns to **design, refine, and evolve governance policies** through meta-reasoning over real-world operational data. |
|
|
| ### The Problem |
|
|
| In modern platforms — social media, enterprise HR, and e-commerce — static policies quickly become outdated or vaguely worded, leading to: |
| - **Inconsistent enforcement**: Moderators interpret "offensive content" differently, creating 300K+ appeals per year (Meta Oversight Board, 2024). |
| - **False-positive actions**: E-commerce platforms lose an estimated $700M/year from incorrectly suspending legitimate high-volume sellers. |
| - **Unaddressed gaps**: Emerging risks like Generative AI misuse have no governing rules in legacy frameworks. |
|
|
| ### The Solution |
|
|
| PolicyEvolverEnv simulates these challenges by presenting the agent with: |
| 1. A **corpus of operational incidents** (flagged posts, HR violations, seller transactions). |
| 2. An **existing policy framework** with known flaws (vague terms, missing rules, conflicting thresholds). |
|
|
| The agent must analyze the data, identify systemic flaws, and submit **structured policy modifications** — not just answers, but actionable governance. The grader evaluates whether the proposed fix is specific, measurable, domain-relevant, and free of hallucination. |
|
|
| ### Why This Matters for RLVR |
|
|
| This environment operates at the **Reinforcement Learning from Verifiable Rewards (RLVR)** layer of inference-time adaptation. No weight updates are performed. The LLM improves within a single 5-step episode by reading its own reward history and staff feedback — demonstrating genuine in-context policy learning. |
|
|
| --- |
|
|
| ## 2. Action Space |
|
|
| The action space uses a **Discriminated Union** (Pydantic `RootModel` with `Discriminator("action_type")`) supporting three structured action types: |
|
|
| ### `propose_clarification` — Easy Task Action |
| |
| | Field | Type | Description | |
| |:------|:-----|:------------| |
| | `action_type` | `Literal["propose_clarification"]` | Discriminator tag | |
| | `ambiguous_term` | `str` | The exact vague term found in existing policies | |
| | `suggested_definition` | `str` | A specific, measurable replacement definition | |
| | `affected_policy_ids` | `List[str]` | Which policy IDs this clarification affects | |
| | `justification` | `str` | Why this term is ambiguous and why the fix works | |
| | `think` | `Optional[str]` | Chain-of-thought reasoning (earns +0.10–0.20 bonus) | |
|
|
| ### `propose_new_rule` — Medium Task Action |
|
|
| | Field | Type | Description | |
| |:------|:-----|:------------| |
| | `action_type` | `Literal["propose_new_rule"]` | Discriminator tag | |
| | `rule_domain` | `str` | Domain the new rule covers (e.g., `"AI_use"`) | |
| | `new_rule` | `str` | The complete new rule text | |
| | `scope` | `List[str]` | Scenario types this rule applies to | |
| | `integration_points` | `List[str]` | How it connects to existing policy IDs | |
| | `justification` | `str` | Why a gap exists and how this rule fills it | |
| | `think` | `Optional[str]` | Chain-of-thought reasoning (earns +0.10–0.20 bonus) | |
|
|
| ### `evolve_policy` — Hard Task Action |
| |
| | Field | Type | Description | |
| |:------|:-----|:------------| |
| | `action_type` | `Literal["evolve_policy"]` | Discriminator tag | |
| | `policy_modifications` | `List[PolicyModification]` | Specific changes: `policy_id`, `change_type`, `new_text`, `reason` | |
| | `expected_outcomes` | `Dict[str, float]` | Metric name → expected value (must show realistic tradeoffs) | |
| | `rollback_conditions` | `List[str]` | When to revert changes | |
| | `justification` | `str` | Comprehensive reasoning for the evolution | |
| | `think` | `Optional[str]` | Chain-of-thought reasoning (earns +0.10–0.20 bonus) | |
|
|
| --- |
|
|
| ## 3. Observation Space |
|
|
| The `Observation` returned by `reset()` and `step()` contains: |
|
|
| | Field | Type | Description | |
| |:------|:-----|:------------| |
| | `task_id` | `str` | Active scenario identifier (`task_easy`, `task_medium`, `task_hard`) | |
| | `episode_id` | `str` | Unique episode session tracker | |
| | `step_count` | `int` | Current step number (max 5 per episode) | |
| | `corpus_size` | `int` | Total incidents in the full data corpus | |
| | `corpus_shown` | `int` | Number of incidents displayed (reactive to agent's domain) | |
| | `data_corpus` | `List[CorpusIncident]` | Operational incidents with `id`, `content`, `system_action`, and `type` | |
| | `current_policies` | `List[Dict]` | The existing policy framework (`id` + `text`) | |
| | `policy_outcomes` | `Optional[List[Dict]]` | Historical outcome data (hard task only) | |
| | `system_metrics` | `Dict[str, float]` | Operational statistics (precision, recall, false-positive rates) | |
| | `identified_issues` | `List[Dict]` | Known flaws in the governance pipeline | |
| | `reward` | `float` | Score from the grader for the last action, in (0, 1) | |
| | `done` | `bool` | Whether the episode has ended | |
| | `info` | `Dict` | Contains `best_score`, `rewards_history`, `steps_remaining`, and `staff_feedback` | |
|
|
| ### Staff Feedback (in `info`) |
|
|
| After each step, the observation includes structured staff feedback to guide the agent's next action: |
|
|
| | Field | Example Values | Purpose | |
| |:------|:---------------|:--------| |
| | `strategic_rating` | `"Junior Associate"`, `"Staff Specialist"`, `"Senior Architect"` | Performance tier based on reward | |
| | `focus` | `"Signal detected"` or `"Burying the lede or distracted by noise"` | Whether the agent prioritized correctly | |
| | `recommendation` | `"Maintain high signal-to-noise ratio and lead with the fix."` | Actionable guidance for next step | |
|
|
| --- |
|
|
| ## 4. Task Descriptions |
|
|
| The environment provides three tasks with escalating cognitive difficulty: |
|
|
| ### Task Easy — Ambiguity Clarification (Difficulty: `easy`) |
| - **Scenario**: A social media platform's community guidelines use vague terms like "offensive" and "appropriate." |
| - **Objective**: Identify an ambiguous term and replace it with a specific, measurable definition. |
| - **Expected Action**: `propose_clarification` |
| - **Expected Min Score**: 0.70 |
| - **Key Grading Criteria**: |
| - Definition must contain measurable keywords (`"threshold"`, `"verify"`, `"%"`, `"within"`) |
| - Vague words (`"generally"`, `"sometimes"`, `"maybe"`) trigger a hard penalty (score capped < 0.30) |
| - Valid `affected_policy_ids` boost score |
|
|
| ### Task Medium — Gap Detection & New Rule (Difficulty: `medium`) |
| - **Scenario**: A corporate HR framework with policies covering data protection but no coverage for Generative AI tool usage. |
| - **Objective**: Detect the missing policy domain and draft a new rule to fill the gap. |
| - **Expected Action**: `propose_new_rule` |
| - **Expected Min Score**: 0.55 |
| - **Key Grading Criteria**: |
| - Must target the correct `rule_domain` (e.g., `"AI_use"`) |
| - Empty `scope` array severely penalized |
| - `integration_points` linking to existing policy IDs boost score |
| - Rule text must be substantive (short rules penalized) |
|
|
| ### Task Hard — Holistic Policy Evolution (Difficulty: `hard`) |
| - **Scenario**: An e-commerce Trust & Safety framework where blanket seller suspension policies catch legitimate seasonal merchants alongside fraudsters. |
| - **Objective**: Evolve multiple policies simultaneously to balance fraud detection, revenue velocity, and seller trust. |
| - **Expected Action**: `evolve_policy` |
| - **Expected Min Score**: 0.40 |
| - **Key Grading Criteria**: |
| - **Hallucination Guard**: All metrics at 0.95+ triggers "Unrealistic Tradeoff" penalty (score capped < 0.15) |
| - **Cross-Domain Guard**: HR/AI proposals for an e-commerce task incur -0.30 penalty |
| - **Realistic Tradeoffs**: `expected_outcomes` must show mathematical variance (improving fraud detection should decrease revenue velocity) |
| - **Domain Relevance**: Modifications must reference marketplace concepts (seller, fraud, listing, merchant) |
| - Metric key aliases supported: `fraud_rate`/`fraud`/`fraud_detection`, `revenue_velocity`/`queue_overload`/`revenue` |
|
|
| ### Global Grading Mechanics (All Tasks) |
|
|
| | Mechanic | Effect | |
| |:---------|:-------| |
| | **Chain-of-Thought Bonus** | `think` field with keywords like `"tradeoff"`, `"precision"`, `"recall"` → +0.10 to +0.20 | |
| | **Step-Delta Bonus** | Significant improvement over previous best → +0.02 to +0.05 | |
| | **Anti-Repetition Penalty** | Exact repeated action → -0.30 | |
| | **Prompt Injection Guard** | `"ignore previous"`, `"system_prompt"`, `"override"` → score zeroed | |
| | **Semantic Density Guard** | Word-stuffing with >200 words and low content density → score zeroed | |
| | **Red Herring Penalty** | Referencing injected noise topics (office logistics, mascot) → up to -0.75 | |
| | **Segmented Prioritization** | Core fix in first 25% of response → bonus; buried at bottom → penalty | |
|
|
| --- |
|
|
| ## 5. Setup and Usage |
|
|
| ### Local Installation |
|
|
| ```bash |
| git clone https://github.com/Luciferai04/PolicyEvolverEnv.git |
| cd PolicyEvolverEnv |
| python3 -m venv .venv |
| source .venv/bin/activate |
| pip install -r server/requirements.txt |
| ``` |
|
|
| ### Run the Environment Server |
|
|
| ```bash |
| uvicorn server.app:app --port 8000 |
| ``` |
|
|
| This starts all endpoints: `/reset` (POST), `/step` (POST), `/state` (GET), `/tasks` (GET), `/grader` (POST), `/health` (GET), `/baseline` (GET). |
|
|
| ### Run with Docker |
|
|
| ```bash |
| docker build -t policy-evolver . |
| docker run -p 8000:8000 policy-evolver |
| ``` |
|
|
| ### Run the Inference Agent |
|
|
| The primary evaluation entry point is `inference.py`, which follows the hackathon `[START]`, `[STEP]`, `[END]` logging format. |
|
|
| ```bash |
| export API_BASE_URL="https://api.groq.com/openai/v1" |
| export MODEL_NAME="llama-3.1-8b-instant" |
| export HF_TOKEN="your_groq_api_key" |
| |
| python3 inference.py |
| ``` |
|
|
| To run a specific task: `python3 inference.py task_easy` |
|
|
| ### Required Environment Variables |
|
|
| | Variable | Description | Example | |
| |:---------|:------------|:--------| |
| | `HF_TOKEN` | API key for LLM inference (Groq) | `gsk_...` | |
| | `API_BASE_URL` | OpenAI-compatible endpoint | `https://api.groq.com/openai/v1` | |
| | `MODEL_NAME` | Model identifier | `llama-3.1-8b-instant` | |
|
|
| ### Run Tests |
|
|
| ```bash |
| PYTHONPATH=. python tests/test_smoke_exploits.py # 27 smoke & exploit checks |
| PYTHONPATH=. python tests/test_icl.py # ICL verification (3 tasks) |
| PYTHONPATH=. python tests/test_multi_episode.py # Multi-episode progression |
| PYTHONPATH=. python server/grader.py # 8-phase grader test suite |
| ``` |
|
|
| --- |
|
|
| ## 6. Baseline Performance Scores |
|
|
| The agent uses **In-Context Reinforcement Learning (ICL-RL)**: no weight updates are performed. The LLM improves within a single 5-step episode by reading its own reward history and staff feedback. |
|
|
| ### Single-Step Convergence (Best Case) |
|
|
| | Task | Score | Converged | Expected Min | |
| |:-----|:------|:----------|:-------------| |
| | `task_easy` | 0.94 | ✓ Step 1 | 0.70 | |
| | `task_medium` | 0.999 | ✓ Step 1 | 0.55 | |
| | `task_hard` | 0.90 | ✓ Step 1 | 0.40 | |
|
|
| ### Multi-Step ICL Progression (Naive → Optimized) |
|
|
| | Task | Naive (Step 0) | Optimized (Step 1) | Improvement | |
| |:-----|:---------------|:-------------------|:------------| |
| | `task_easy` | 0.400 | 0.999 | +0.600 | |
| | `task_medium` | 0.001 | 0.999 | +0.998 | |
| | `task_hard` | 0.088 | 0.999 | +0.912 | |
|
|
| **Average ICL Improvement: +0.837** |
|
|
| ### Configuration |
|
|
| | Setting | Value | |
| |:--------|:------| |
| | **Model** | `llama-3.1-8b-instant` (via Groq) | |
| | **Temperature** | `0.0` | |
| | **Seed** | `42` | |
| | **Determinism** | 5 identical runs → identical scores ✓ | |
| | **Fine-tuning** | None required | |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ``` |
| policy_evolver_env/ |
| ├── inference.py # Hackathon entry point ([START]/[STEP]/[END] format) |
| ├── client.py # EnvClient for HTTP interaction |
| ├── models.py # Pydantic models (Action, Observation, State) |
| ├── openenv.yaml # OpenEnv specification |
| ├── Dockerfile # Docker deployment with HEALTHCHECK |
| ├── server/ |
| │ ├── app.py # FastAPI + Gradio dashboard |
| │ ├── environment.py # Environment logic (reset, step, state) |
| │ ├── grader.py # Deterministic grading engine (8-phase test suite) |
| │ ├── requirements.txt # Dependencies |
| │ └── tasks/ # Task definitions (easy, medium, hard) |
| ├── tests/ |
| │ ├── test_smoke_exploits.py # 27 smoke & exploit checks |
| │ ├── test_icl.py # ICL loop verification |
| │ └── test_multi_episode.py # Multi-episode progression |
| └── STRATEGIC_LEARNING.md # RLVR architecture documentation |
| ``` |
|
|
