restructured inference.py and updated readme

README.md

@@ -1,41 +1,68 @@
+---
+title: Scheduling Assistant OpenEnv
+emoji: 🗓️
+colorFrom: blue
+colorTo: purple
+sdk: docker
+pinned: false
+tags:
+  - openenv
+  - benchmark
+  - agents
+---
+
 # Scheduling Assistant - OpenEnv
 
-A simulated corporate scheduling backend. The agent acts as a resource coordinator managing calendars, timezone math, and meeting conflicts.
+## Motivation & Description
+Corporate scheduling is notoriously difficult for LLM agents. While static QA benchmarks test reasoning in a vacuum, calendar management evaluates an agent's ability to act on temporal data, perform cross-timezone arithmetic, and navigate multi-step constraint satisfaction. This OpenEnv benchmark simulates a backend scheduling system to rigidly test these capabilities in a programmatic, autonomous evaluation loop.
 
-## Overview
-This environment evaluates an agent's ability to:
-1. Lookup employee information (names, timezones)
-2. View cross-timezone calendars and manage overlap.
-3. Successfully book and prioritize syncs across different levels of complexity.
+## Setup & Running
+The environment is containerized. It should run seamlessly via Docker or bare metal Python.
 
-Strict constraints:
-- Built purely in Python without external databases to comply with 2 vCPU and 8GB RAM restrictions on Hugging Face Spaces.
+### Via Bare Metal:
+```bash
+cd scheduling_assistant
+pip install -r requirements.txt
+export OPENAI_API_KEY="sk-..."       # Or any valid proxy like Groq
+export TASK_LEVEL="medium"           # "easy" | "medium" | "hard"
+python inference.py
+```
 
-## Task Difficulties (`TASK_LEVEL`)
-- **Easy**: Book a 30-minute sync between two people in the same timezone (UTC).
-- **Medium**: Schedule a 1-hour global all-hands matching the 9-to-5 working hours of 4 employees across PST, EST, UTC, and IST.
-- **Hard**: Override a low priority meeting to book a last-minute VIP event, enforcing priority logic without cancelling critical syncs.
+### Via Docker:
+```bash
+docker build -t openenv-scheduler .
+docker run -e OPENAI_API_KEY="sk-..." -e TASK_LEVEL="easy" openenv-scheduler
+```
 
 ## Action Space
-`Action` expects a JSON object containing:
-- `action_type`: One of `['lookup_employee', 'view_calendar', 'book_meeting', 'cancel_meeting', 'submit_task']`
-- `employee_ids`: Array of strings
-- `start_time`: ISO 8601 string
-- `end_time`: ISO 8601 string
-- `meeting_id`: String (uuid)
+The agent navigates the environment using the `step_environment` tool. The action schema expects a JSON object containing:
+- `action_type`: Evaluated string routing. One of `['lookup_employee', 'view_calendar', 'book_meeting', 'cancel_meeting', 'submit_task']`
+- `employee_ids`: Array of strings (e.g. `["charlie", "alice"]`)
+- `start_time`: ISO 8601 string containing timezone information
+- `end_time`: ISO 8601 string containing timezone information
+- `meeting_id`: String (UUID) targeted for cancellation
 
 ## Observation Space
-`Observation` returns:
-- `current_simulated_time`: ISO 8601
-- `task_description`: Instructions for the episode
-- `last_action_result`: Outcome of the previous tool call
-- `error_message`: Formatting limitations or failed constraint messages
+The environment emits a strict observation dictionary after every action:
+- `current_simulated_time`: Static simulated execution time in ISO 8601.
+- `task_description`: The specific instructions and targets for the chosen episode difficulty.
+- `last_action_result`: Outcome array/string of the previously called functionality (e.g. returned calendars).
+- `error_message`: Missing variables, formatting limitations, or failed logical constraint messages (e.g. "Outside working hours for {employee}").
 
-## Setup & Running
+## Task Descriptions & Difficulty
 
-```bash
-pip install -r requirements.txt
-export OPENAI_API_KEY="sk-..."
-export TASK_LEVEL="hard" # "easy" | "medium" | "hard"
-python inference.py
-```
+- **Easy**: Book a 30-minute sync between two people in the identical timezone (UTC) tomorrow. 
+  - *Evaluation*: Tests basic JSON array filtering and time addition within standard constraints.
+- **Medium**: Schedule a 1-hour global all-hands matching the active 9-to-5 working hours of 4 specific employees across distinct timezones (PST, EST, UTC, and CST).
+  - *Evaluation*: Tests rigorous temporal overlap arithmetic. There is exactly one valid 1-hour global window where all 4 conditions are met. 
+- **Hard**: The CEO needs an urgent sync that overlaps a booked slot. The agent must override a "low priority" internal meeting to book the VIP event, and reschedule the bumped meeting, all without cancelling "high priority" syncs.
+  - *Evaluation*: Tests multi-hop logic, priority understanding, and state backtracking over multiple continuous actions.
+
+## Baseline Scores
+Testing was conducted locally with **`llama-3.3-70b-versatile`** (Groq) heavily capped by `MAX_STEPS = 15`.
+
+| Difficulty | Baseline Score (Total Reward) | Observations |
+| :--- | :--- | :--- |
+| **Easy** | `1.0` | Successfully deduced calendar holes, passed the constraint checker, and successfully booked. |
+| **Medium** | `0.2` | Successfully filtered out the employee IDs and scanned calendars, but critically failed to blindly deduce the precise global hour overlap. Trapped in an iterative trial-and-error cycle until max steps were reached. |
+| **Hard** | `0.1 - 0.2` | Agent correctly maps the target IDs but is mathematically bottlenecked similar to the Medium benchmark. Exhausts tools prior to completion. |

scheduling_assistant/env.py → env.py

@@ -23,7 +23,7 @@ EMPLOYEES = {
     "bob": {"id": "bob", "name": "Bob", "timezone": "UTC"},
     "charlie": {"id": "charlie", "name": "Charlie", "timezone": "US/Pacific"},
     "dave": {"id": "dave", "name": "Dave", "timezone": "US/Eastern"},
-    "eve": {"id": "eve", "name": "Eve", "timezone": "Asia/Kolkata"},
+    "eve": {"id": "eve", "name": "Eve", "timezone": "US/Central"},
     "ceo": {"id": "ceo", "name": "CEO", "timezone": "US/Eastern"},
     "vp_sales": {"id": "vp_sales", "name": "VP of Sales", "timezone": "US/Pacific"}
 }
@@ -73,7 +73,7 @@ class SchedulingEnv:
             
         elif self.task_level == "medium":
             self.task_description = (
-                "Schedule a 1-hour meeting for 4 people: Charlie (PST), Dave (EST), Alice (UTC), and Eve (IST). "
+                "Schedule a 1-hour meeting for 4 people: Charlie (PST), Dave (EST), Alice (UTC), and Eve (CST). "
                 "The meeting must fall specifically within the 9-to-5 local working hours of ALL 4 participants tomorrow (Oct 11). "
                 "When finished, run submit_task."
             )
@@ -161,6 +161,8 @@ class SchedulingEnv:
         return False
 
     def step(self, action: Action) -> tuple[Dict, float, bool]:
+        if action.employee_ids:
+            action.employee_ids = [e.lower() for e in action.employee_ids]
         self.current_step += 1
         reward = 0.0
         done = False

scheduling_assistant/inference.py → inference.py

@@ -1,7 +1,28 @@
 import os
 import json
+from typing import List, Optional
 from openai import OpenAI
 from env import SchedulingEnv, Action
+from dotenv import load_dotenv
+
+load_dotenv()
+
+def log_start(task: str, env: str, model: str) -> None:
+    print(f"[START] task={task} env={env} model={model}", flush=True)
+
+def log_step(step: int, action: str, reward: float, done: bool, error: Optional[str]) -> None:
+    error_val = str(error).replace('\n', ' ') if error else "null"
+    done_val = str(done).lower()
+    # clean newlines from action
+    action = action.replace('\n', ' ')
+    print(
+        f"[STEP] step={step} action={action} reward={reward:.2f} done={done_val} error={error_val}",
+        flush=True,
+    )
+
+def log_end(success: bool, steps: int, score: float, rewards: List[float]) -> None:
+    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
+    print(f"[END] success={str(success).lower()} steps={steps} score={score:.3f} rewards={rewards_str}", flush=True)
 
 def main():
     hf_token = os.getenv("HF_TOKEN")
@@ -29,7 +50,7 @@ def main():
     
     obs = env.reset()
     
-    print("[START]")
+    log_start(task=task_level, env="scheduling_benchmark", model=model_name)
     
     messages = [
         {
@@ -61,18 +82,26 @@ def main():
         }
     ]
 
+    rewards_list = []
     total_reward = 0.0
+    steps_taken = 0
+    done = False
     
     for step_num in range(1, env.max_steps + 1):
-        print(f"[STEP] {step_num}")
-        
-        response = client.chat.completions.create(
-            model=model_name,
-            messages=messages,
-            tools=tools,
-            tool_choice={"type": "function", "function": {"name": "step_environment"}}
-        )
+        steps_taken = step_num
         
+        try:
+            response = client.chat.completions.create(
+                model=model_name,
+                messages=messages,
+                tools=tools,
+                tool_choice={"type": "function", "function": {"name": "step_environment"}}
+            )
+        except Exception as e:
+            error_msg = f"Model API Error: {str(e)}"
+            log_step(step=step_num, action="api_call_failed", reward=total_reward, done=True, error=error_msg)
+            break
+            
         response_message = response.choices[0].message
         messages.append(response_message)
         
@@ -80,17 +109,22 @@ def main():
         if not tool_calls:
             # Model failed to call a tool, force a submit task
             action = Action(action_type="submit_task")
+            action_str = "submit_task (force_fallback)"
             obs, reward, done = env.step(action)
             total_reward = reward
+            rewards_list.append(reward)
+            log_step(step=step_num, action=action_str, reward=reward, done=done, error=obs['error_message'])
             break
             
         tool_call = tool_calls[0]
+        action_args_str = tool_call.function.arguments
         try:
-            action_args = json.loads(tool_call.function.arguments)
+            action_args = json.loads(action_args_str)
             action = Action(**action_args)
         except Exception as e:
             # Fallback if invalid action
             action = Action(action_type="submit_task")
+            action_args_str = f"INVALID_JSON: {action_args_str}"
             
         messages.append({
             "role": "tool",
@@ -101,16 +135,21 @@ def main():
         
         obs, reward, done = env.step(action)
         total_reward = reward
+        rewards_list.append(reward)
+        
+        # log step as per stdout rules
+        log_step(step=step_num, action=action_args_str, reward=reward, done=done, error=obs.get('error_message'))
         
+        if done:
+            break
+            
         messages.append({
             "role": "user",
             "content": f"Observation:\n{json.dumps(obs)}\nReward so far: {reward}"
         })
-        
-        if done:
-            break
 
-    print(f"[END] Final Reward: {total_reward}")
+    success = total_reward >= 1.0
+    log_end(success=success, steps=steps_taken, score=total_reward, rewards=rewards_list)
 
 if __name__ == "__main__":
     main()

scheduling_assistant/requirements.txt → requirements.txt

@@ -2,3 +2,4 @@ pydantic
 openai
 python-dateutil
 pytz
+dotenv