README.md

metadata

title: Eco-LLM Inference Routing Environment
emoji: 🌱
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
tags:
  - openenv
  - reinforcement-learning
  - llm-routing
  - carbon-aware
  - systems

Eco-LLM Inference Routing Environment

Eco-LLM Inference Routing is an OpenEnv-compatible reinforcement learning environment for carbon-aware LLM serving. The agent must decide how to route each query across model tiers and routing strategies while balancing answer quality, latency, energy use, and grid carbon intensity.

Public Space: https://huggingface.co/spaces/rudhratej09/Eco-LLM-Carbon-Aware-LLM-Inference-Routing

after running the space for ui: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/ui/

Live endpoints:

• /tasks: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/tasks
• /metadata: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/metadata
• /docs: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/docs

Why This Environment

Modern LLM systems do not have a single objective. A router often has to decide whether a request should go to a small model, a larger model, a cache, a retrieval layer, or be delayed until carbon intensity improves. This environment turns that operational tradeoff into a deterministic RL benchmark with structured rewards and graded tasks.

The design stays close to a real serving stack:

• multiple model tiers with different energy and latency costs
• a cache that can eliminate repeat inference cost
• a knowledge-base path for FAQ-style requests
• cascade and early-exit strategies for adaptive compute
• a carbon schedule that makes timing matter

Tasks

Three tasks are included, each with a deterministic grader returning a score in [0.0, 1.0].

• task_1: single-query routing with a simple accuracy-versus-cost tradeoff
• task_2: three-query episode with an explicit penalty for overusing the large model
• task_3: five-query stateful episode with repeated queries, cache reuse, knowledge-base lookups, cascading, and carbon-aware waiting

Task definitions live in server/tasks.py.

Action Space

Defined in server/models.py.

Each action has:

• strategy: NONE, USE_CACHE, DO_CASCADE, EARLY_EXIT, WAIT, or CALL_KB
• model_choice: SMALL, MEDIUM, or LARGE
• exit_flag: whether a cascade should stop early after the first correct answer

Observation Space

Each observation includes:

• current query text
• current grid carbon intensity
• cache contents
• whether a knowledge base is available
• scalar reward
• detailed reward breakdown
• done flag
• task and state metadata

Reward Design

The environment is dense-reward, not purely terminal.

Per step:

• score = 1.0 when the chosen strategy reaches a model capable of solving the query, else 0.0
• carbon_penalty = carbon_intensity * energy_cost, capped at 0.8
• latency_penalty = 0.1 * latency_cost
• cache hits add +0.5
• successful early exit adds +0.1
• some tasks add an explicit large-model penalty

Model costs:

Model	Energy cost	Latency cost
SMALL	0.1	1.0
MEDIUM	0.3	2.0
LARGE	0.6	5.0

This makes the benchmark about policy quality, not just raw correctness.

Core Files

• openenv.yaml: OpenEnv manifest
• server/app.py: FastAPI app and HTTP endpoints
• server/env.py: environment logic
• server/models.py: typed action, observation, and state models
• server/tasks.py: deterministic task catalog and cost tables
• server/rubrics.py: rubric-style graders
• graders.py: top-level grader compatibility shims
• inference.py: root-level baseline runner
• baseline.py: random, heuristic, and LLM baselines

Local Run

Install dependencies:

uv sync

Start the server:

uvicorn server.app:app --host 0.0.0.0 --port 7860

Quick checks:

curl http://127.0.0.1:7860/health
curl http://127.0.0.1:7860/metadata
curl http://127.0.0.1:7860/tasks

Baseline Inference

The required root-level baseline script is inference.py.

Example:

export HF_TOKEN=your_token
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct
export ECO_LLM_TASK=task_3
python inference.py

The script emits OpenEnv-style logs:

[START] task=task_3 env=eco_llm_inference_routing model=Qwen/Qwen2.5-72B-Instruct
[STEP] step=1 action=strategy=CALL_KB,model=SMALL,exit=false reward=1.30 done=false error=null
[END] success=true steps=5 score=0.72 rewards=1.30,0.48,1.50,0.20,1.90

Baselines

The repository includes three reference agents:

• random policy
• heuristic router
• LLM-based router

Run comparison:

python baseline.py compare --task task_3 --episodes 20

These baselines are useful because they show the environment is not just a one-step classification toy. Cache-aware and carbon-aware decisions materially change returns on the harder task.

API Surface

Main endpoints:

• GET /
• GET /health
• GET /metadata
• GET /schema
• GET /tasks
• GET /tasks/meta
• GET /tasks/{task_id}
• POST /reset
• POST /step
• POST /grader
• POST /grade
• POST /grader/replay
• GET /docs

OpenEnv Compatibility

This submission includes the expected OpenEnv pieces:

• openenv.yaml
• root-level inference.py
• root-level task and grader compatibility exports
• typed action, observation, and state models
• Docker runtime for Hugging Face Spaces deployment

Docker

Build:

docker build -t eco-llm-routing .

Run:

docker run -p 7860:7860 eco-llm-routing

Validation Notes

The repository includes lightweight tests in tests/test_agents.py. For a submission review, the most important checks are:

• server boots cleanly
• /tasks, /metadata, and /docs resolve
• the grader returns normalized scores
• the root-level inference script remains runnable

License

BSD-style license. See LICENSE.