Spaces:
Runtime error
Runtime error
| title: Eco-LLM Inference Routing Environment | |
| emoji: "🌱" | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| tags: | |
| - openenv | |
| - reinforcement-learning | |
| - llm-routing | |
| - carbon-aware | |
| - systems | |
| # Eco-LLM Inference Routing Environment | |
| Eco-LLM Inference Routing is an OpenEnv-compatible reinforcement learning environment for carbon-aware LLM serving. The agent must decide how to route each query across model tiers and routing strategies while balancing answer quality, latency, energy use, and grid carbon intensity. | |
| Public Space: | |
| https://huggingface.co/spaces/rudhratej09/Eco-LLM-Carbon-Aware-LLM-Inference-Routing | |
| after running the space for ui: | |
| https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/ui/ | |
| Live endpoints: | |
| - `/tasks`: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/tasks | |
| - `/metadata`: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/metadata | |
| - `/docs`: https://rudhratej09-eco-llm-carbon-aware-llm-inference-routing.hf.space/docs | |
| ## Why This Environment | |
| Modern LLM systems do not have a single objective. A router often has to decide whether a request should go to a small model, a larger model, a cache, a retrieval layer, or be delayed until carbon intensity improves. This environment turns that operational tradeoff into a deterministic RL benchmark with structured rewards and graded tasks. | |
| The design stays close to a real serving stack: | |
| - multiple model tiers with different energy and latency costs | |
| - a cache that can eliminate repeat inference cost | |
| - a knowledge-base path for FAQ-style requests | |
| - cascade and early-exit strategies for adaptive compute | |
| - a carbon schedule that makes timing matter | |
| ## Tasks | |
| Three tasks are included, each with a deterministic grader returning a score in `[0.0, 1.0]`. | |
| - `task_1`: single-query routing with a simple accuracy-versus-cost tradeoff | |
| - `task_2`: three-query episode with an explicit penalty for overusing the large model | |
| - `task_3`: five-query stateful episode with repeated queries, cache reuse, knowledge-base lookups, cascading, and carbon-aware waiting | |
| Task definitions live in [server/tasks.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/tasks.py). | |
| ## Action Space | |
| Defined in [server/models.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/models.py). | |
| Each action has: | |
| - `strategy`: `NONE`, `USE_CACHE`, `DO_CASCADE`, `EARLY_EXIT`, `WAIT`, or `CALL_KB` | |
| - `model_choice`: `SMALL`, `MEDIUM`, or `LARGE` | |
| - `exit_flag`: whether a cascade should stop early after the first correct answer | |
| ## Observation Space | |
| Each observation includes: | |
| - current query text | |
| - current grid carbon intensity | |
| - cache contents | |
| - whether a knowledge base is available | |
| - scalar reward | |
| - detailed reward breakdown | |
| - done flag | |
| - task and state metadata | |
| ## Reward Design | |
| The environment is dense-reward, not purely terminal. | |
| Per step: | |
| - `score = 1.0` when the chosen strategy reaches a model capable of solving the query, else `0.0` | |
| - `carbon_penalty = carbon_intensity * energy_cost`, capped at `0.8` | |
| - `latency_penalty = 0.1 * latency_cost` | |
| - cache hits add `+0.5` | |
| - successful early exit adds `+0.1` | |
| - some tasks add an explicit large-model penalty | |
| Model costs: | |
| | Model | Energy cost | Latency cost | | |
| |---|---:|---:| | |
| | `SMALL` | 0.1 | 1.0 | | |
| | `MEDIUM` | 0.3 | 2.0 | | |
| | `LARGE` | 0.6 | 5.0 | | |
| This makes the benchmark about policy quality, not just raw correctness. | |
| ## Core Files | |
| - [openenv.yaml](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/openenv.yaml): OpenEnv manifest | |
| - [server/app.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/app.py): FastAPI app and HTTP endpoints | |
| - [server/env.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/env.py): environment logic | |
| - [server/models.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/models.py): typed action, observation, and state models | |
| - [server/tasks.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/tasks.py): deterministic task catalog and cost tables | |
| - [server/rubrics.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/server/rubrics.py): rubric-style graders | |
| - [graders.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/graders.py): top-level grader compatibility shims | |
| - [inference.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/inference.py): root-level baseline runner | |
| - [baseline.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/baseline.py): random, heuristic, and LLM baselines | |
| ## Local Run | |
| Install dependencies: | |
| ```bash | |
| uv sync | |
| ``` | |
| Start the server: | |
| ```bash | |
| uvicorn server.app:app --host 0.0.0.0 --port 7860 | |
| ``` | |
| Quick checks: | |
| ```bash | |
| curl http://127.0.0.1:7860/health | |
| curl http://127.0.0.1:7860/metadata | |
| curl http://127.0.0.1:7860/tasks | |
| ``` | |
| ## Baseline Inference | |
| The required root-level baseline script is [inference.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/inference.py). | |
| Example: | |
| ```bash | |
| export HF_TOKEN=your_token | |
| export API_BASE_URL=https://router.huggingface.co/v1 | |
| export MODEL_NAME=Qwen/Qwen2.5-72B-Instruct | |
| export ECO_LLM_TASK=task_3 | |
| python inference.py | |
| ``` | |
| The script emits OpenEnv-style logs: | |
| ```text | |
| [START] task=task_3 env=eco_llm_inference_routing model=Qwen/Qwen2.5-72B-Instruct | |
| [STEP] step=1 action=strategy=CALL_KB,model=SMALL,exit=false reward=1.30 done=false error=null | |
| [END] success=true steps=5 score=0.72 rewards=1.30,0.48,1.50,0.20,1.90 | |
| ``` | |
| ## Baselines | |
| The repository includes three reference agents: | |
| - random policy | |
| - heuristic router | |
| - LLM-based router | |
| Run comparison: | |
| ```bash | |
| python baseline.py compare --task task_3 --episodes 20 | |
| ``` | |
| These baselines are useful because they show the environment is not just a one-step classification toy. Cache-aware and carbon-aware decisions materially change returns on the harder task. | |
| ## API Surface | |
| Main endpoints: | |
| - `GET /` | |
| - `GET /health` | |
| - `GET /metadata` | |
| - `GET /schema` | |
| - `GET /tasks` | |
| - `GET /tasks/meta` | |
| - `GET /tasks/{task_id}` | |
| - `POST /reset` | |
| - `POST /step` | |
| - `POST /grader` | |
| - `POST /grade` | |
| - `POST /grader/replay` | |
| - `GET /docs` | |
| ## OpenEnv Compatibility | |
| This submission includes the expected OpenEnv pieces: | |
| - `openenv.yaml` | |
| - root-level `inference.py` | |
| - root-level task and grader compatibility exports | |
| - typed action, observation, and state models | |
| - Docker runtime for Hugging Face Spaces deployment | |
| ## Docker | |
| Build: | |
| ```bash | |
| docker build -t eco-llm-routing . | |
| ``` | |
| Run: | |
| ```bash | |
| docker run -p 7860:7860 eco-llm-routing | |
| ``` | |
| ## Validation Notes | |
| The repository includes lightweight tests in [tests/test_agents.py](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/tests/test_agents.py). For a submission review, the most important checks are: | |
| - server boots cleanly | |
| - `/tasks`, `/metadata`, and `/docs` resolve | |
| - the grader returns normalized scores | |
| - the root-level inference script remains runnable | |
| ## License | |
| BSD-style license. See [LICENSE](/C:/Users/Rudhratej/CodeMara-Meta-OpenEnv-Hackathon/LICENSE). | |