README.md

---

title: Store Manager Environment
emoji: 🛒
colorFrom: blue
colorTo: yellow
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
---


# Store Manager Environment

A retail store simulator where an LLM agent acts as the store manager. The agent observes full inventory state each step and chooses one action — discount a product, restock inventory, or move a product to a better shelf zone — to influence customer demand and maximize cumulative profit over a fixed episode horizon.

---

## What Makes This Environment Interesting

This environment models three interconnected real-world retail mechanics that require the agent to reason about **compounding consequences**:

- **Discounts are permanent within an episode.** Every discount cuts the current (already-discounted) price further. A 20% discount applied twice yields 36% off the original price. The agent must decide how aggressively to discount — deep cuts clear stock fast but permanently erode margin.
- **Restocking has a time lag.** Stock doesn't arrive instantly. Each product has a `restock_lead_time` (1–3 steps). The agent must plan ahead — ordering too late leaves the shelf empty; ordering too much risks expiry penalties and inflated holding costs.
- **Zone placement directly multiplies pick probability.** Zone 1 (premium) boosts visibility 1.5×, but only holds 2 products. The agent must continuously rotate which products deserve premium placement.

These three mechanics combine into a rich multi-step decision problem where each action affects future state: a discount today changes tomorrow's revenue; a restock today affects holding costs for the next several steps; a zone move today costs $3 but can double a product's sell-through rate.

---

## Task Overview

| Component | Description |
|---|---|
| **Products** | N products, each with cost price, selling price, quantity, expiry step, shelf zone, and a base pick probability |
| **Customers** | Fixed number of customers per step pick products according to a normalized probability distribution |
| **Agent Action** | Exactly one of three action types per step: `discount`, `restock`, or `place` |
| **Reward** | Normalized sigmoid reward in `[0.0, 1.0]` based on step profit vs. per-step target |
| **Expiry penalty** | `remaining_quantity × cost_price` charged when a product reaches its expiry step |
| **Episode end** | After `max_steps` steps, or when all products are inactive |

### Pick Probability Formula

Customer pick probability combines zone placement and discount multiplier:

```

effective_weight[i] = base_probability[i] × zone_multiplier[zone[i]] × (1 + discount_pct[i] / 100)

pick_prob[i]        = effective_weight[i] / sum(effective_weight for all active products)

```

---

## Actions

The agent submits exactly one action per step.

### 1. Discount

Reduce the selling price of a product. Discounts **compound across steps within an episode** — each discount cuts the current price further, not the original base price.

```json

{ "action_type": "discount", "product_id": 2, "discount_pct": 20 }

```

| Field | Type | Values |
|---|---|---|
| `action_type` | string | `"discount"` |
| `product_id` | int | any valid product ID |
| `discount_pct` | int | `0`, `10`, `20`, or `50` |

**Compounding example** (base price $10.00):

| Step | Discount | Selling Price | Total off base |
|------|----------|--------------|----------------|
| 1 | 20% | $8.00 | −20% |
| 2 | 20% | $6.40 | −36% |
| 3 | 20% | $5.12 | −49% |
| 4 | 50% | $2.56 | −74% |

- `base_selling_price` is never modified — it serves as a reference for how far prices have been cut.
- `current_discount_pct` reflects only the discount tier applied **this step** (used in pick-probability formula).
- Prices reset to `base_selling_price` at the start of each new episode (`reset()`).
- A `0%` discount leaves the price unchanged but still advances the step.

**When to use:** Apply deep discounts to near-expiry stock to clear inventory before the expiry penalty hits. Use light discounts on high-margin products to drive volume without destroying margin.

---

### 2. Restock

Place a purchase order for more units of a product. Stock arrives after `restock_lead_time` steps. The ordering cost is charged **immediately** this step.

```json

{ "action_type": "restock", "product_id": 1, "restock_quantity": 15 }

```

| Field | Type | Values |
|---|---|---|
| `action_type` | string | `"restock"` |
| `product_id` | int | any valid product ID |
| `restock_quantity` | int | any positive integer |

**How it works:**
- On restock: `pending_restock_quantity = restock_quantity`, `pending_restock_arrives_at_step = current_step + restock_lead_time`
- When delivery step is reached: `quantity += restock_quantity`, pending fields cleared
- Only **one pending order per product** at a time — wait for delivery before reordering

**Costs:**
- `restock_cost = restock_quantity × restock_cost_per_unit` charged upfront this step
- `holding_cost = quantity × cost_price × holding_cost_rate` charged every step on all stock regardless of action

**Per-product restock parameters (randomised at episode start):**

| Field | Range | Description |
|---|---|---|
| `restock_lead_time` | 1–3 steps | Steps until delivery |
| `restock_cost_per_unit` | `cost_price × 1.00–1.25` | Ordering cost per unit |
| `holding_cost_rate` | 0.01–0.04 | Fraction of `cost_price` charged per unit per step |

**Error cases:** Invalid if `restock_quantity ≤ 0` or product already has a pending order.

**When to use:** Replenish a fast-selling product before it stocks out, only when enough steps remain to sell through the new inventory before expiry.

---

### 3. Place (Zone Placement)

Move a product to a different shelf zone. Takes effect immediately. Charges a flat **labor fee of $3.00** per move.

```json

{ "action_type": "place", "product_id": 3, "target_zone": 1 }

```

| Field | Type | Values |
|---|---|---|
| `action_type` | string | `"place"` |
| `product_id` | int | any valid product ID |
| `target_zone` | int | `1`, `2`, or `3` |

**Zone multipliers and capacity:**

| Zone | Location | Pick Probability Multiplier | Max Products |
|------|----------|-----------------------------|--------------|
| 1 | Premium / entrance / eye-level | **×1.5** | 2 |
| 2 | Standard shelves | **×1.0** | 4 |
| 3 | Back / low-traffic | **×0.6** | unlimited |

**Initial zone assignment at episode start:** products fill zone 1 first (up to capacity 2), then zone 2, remainder in zone 3.

**Error cases:** Invalid if `target_zone` is not in {1, 2, 3}, product is already in `target_zone`, or target zone is at capacity.

**When to use:** Rotate zone 1 to whichever product most needs visibility — near-expiry items benefit most since clearance speed directly avoids penalty. Demote slow-movers to zone 3 to free up premium space.

---

## Reward Function

Each step returns a **normalized reward in `[0.0, 1.0]`** using a sigmoid centred at break-even:

```

step_profit = step_revenue − cost_of_goods_sold − expiry_penalty

            − holding_cost − placement_cost − restock_cost



x      = step_profit / per_step_target

reward = sigmoid(2x) = 1 / (1 + exp(−2x))

```

| Step profit vs. target | reward |
|---|---|
| Catastrophic loss (−∞) | → 0.0 |
| Break-even (0) | = 0.50 |
| On target | ≈ 0.88 |
| 2× target | ≈ 0.98 |

The raw per-step cost breakdown is available in `metadata`:

```json

{

  "step_revenue": 45.20,

  "step_cost": 18.40,

  "expiry_penalty": 0.0,

  "holding_cost": 12.35,

  "placement_cost": 3.0,

  "restock_cost": 0.0,

  "step_profit": 11.45

}

```

---

## Difficulty Tasks

Four pre-configured tasks with increasing complexity. Each task has a fixed seed for deterministic, reproducible grading.

| Task | Products | Steps | Customers/step | Expiry range | Profit target | Dominant mechanic |
|------|----------|-------|----------------|--------------|--------------|-------------------|
| **easy** | 4 | 20 | 15 | 18–30 steps | $120 | Discount |
| **medium** | 8 | 28 | 12 | 12–22 steps | $200 | Discount + zone placement |
| **hard** | 10 | 30 | 10 | 8–18 steps | $240 | Discount + zone + expiry timing |
| **expert** | 12 | 25 | 8 | 4–10 steps | $190 | All three actions coordinated |

### Task Descriptions

**easy** — Products expire well after the episode ends; no expiry pressure. The agent learns to identify high-margin products and apply targeted discounts to boost their pick probability.

**medium** — Several products expire mid-episode. Zone 1 has only 2 slots for 8 competing products — the agent must combine discounts with smart zone rotation to clear stock before expiry.

**hard** — Most products expire in 8–18 steps. Expiry management is critical: rotate zone 1 frequently to accelerate clearance, and consider restocking fast-sellers that run out early.

**expert** — Aggressive expiry (4–10 steps) with limited initial stock. All three actions must be coordinated: discount near-expiry items heavily, restock high-margin products before they stock out, and rotate zone 1 continuously.

### Grading

```

score = clamp(cumulative_profit / profit_target, 0.0, 1.0)

```

---

## Baseline Scores

Expected performance using `gpt-4o-mini` (deterministic, fixed seeds):

| Task | Profit Target | Expected Profit | Expected Score |
|------|--------------|-----------------|----------------|
| easy | $120 | ~$140–180 | ~0.90–1.0 |
| medium | $200 | ~$160–220 | ~0.80–1.0 |
| hard | $240 | ~$180–260 | ~0.75–1.0 |
| expert | $190 | ~$100–170 | ~0.50–0.90 |

Scores are fully reproducible. Re-run `inference.py` with the same model and seeds to verify.

---

## Quick Start

```python

from StoreManager import StoremanagerAction, StoremanagerEnv



with StoremanagerEnv(base_url="http://localhost:8000") as env:

    result = env.reset()

    print([p.name for p in result.observation.inventory])



    # Discount: compounds on current price each time applied

    result = env.step(StoremanagerAction(action_type="discount", product_id=0, discount_pct=20))

    print(f"Reward: {result.reward:.2f}")



    # Restock: stock arrives after restock_lead_time steps

    result = env.step(StoremanagerAction(action_type="restock", product_id=1, restock_quantity=10))

    print(f"Restock cost: {result.observation.restock_cost_this_step:.2f}")



    # Place: move product to premium zone (costs $3.00 labor)

    result = env.step(StoremanagerAction(action_type="place", product_id=2, target_zone=1))

    print(f"Placement cost: {result.observation.placement_cost_this_step:.2f}")

    print(f"Zone occupancy: {result.observation.zone_occupancy}")

```

---

## Running Locally

```bash

# Install dependencies

uv sync



# Run dev server (auto-reload, default config)

uvicorn server.app:app --reload



# Run with a specific difficulty task baked in at startup

STORE_TASK=hard uvicorn server.app:app --reload



# Test environment logic directly (no server needed)

python server/StoreManager_environment.py



# Run the WebSocket test suite (server must be running)

python test_ws.py

```

---

## Running the Inference Script

The `inference.py` script runs an LLM agent through all four tasks sequentially and reports a score for each.

### Required Environment Variables

| Variable | Description |
|---|---|
| `HF_TOKEN` | API key — your Hugging Face token or OpenAI API key |
| `API_BASE_URL` | LLM API base URL (e.g. `https://api-inference.huggingface.co/v1/`) |
| `MODEL_NAME` | Model identifier (e.g. `meta-llama/Llama-3.1-8B-Instruct` or `gpt-4o-mini`) |

```bash

# Server must be running first

uvicorn server.app:app &



# Run with Hugging Face Inference API

HF_TOKEN=hf_... \

API_BASE_URL=https://api-inference.huggingface.co/v1/ \

MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct \

python inference.py



# Run with OpenAI (API_BASE_URL defaults to OpenAI when unset)

HF_TOKEN=sk-... MODEL_NAME=gpt-4o-mini python inference.py



# Override the environment server URL

HF_TOKEN=sk-... MODEL_NAME=gpt-4o-mini STORE_ENV_URL=http://localhost:8000 python inference.py

```

### Structured Log Format

```

[START] {"task": "easy", "seed": 42, "model": "gpt-4o-mini", "max_steps": 20, "profit_target": 120.0}

[STEP]  {"task": "easy", "step": 1, "action": {"action_type": "discount", "product_id": 2, "discount_pct": 20}, "reward": 0.88, "cumulative_profit": 12.5, "holding_cost": 8.2, "placement_cost": 0, "restock_cost": 0}

...

[END]   {"task": "easy", "cumulative_profit": 145.3, "score": 1.0, "profit_target": 120.0}

```

A single server instance handles all four tasks — the task config is passed in the `reset` message, so no server restart is needed between tasks.

---

## Action & Observation Reference

### Action: `StoremanagerAction`

| Field | Type | Default | Description |
|---|---|---|---|
| `action_type` | `"discount"` \| `"restock"` \| `"place"` | `"discount"` | Type of action |
| `product_id` | int | required | ID of the product to act on |
| `discount_pct` | `0` \| `10` \| `20` \| `50` | `0` | Discount tier (for `discount` actions) |
| `restock_quantity` | int | `None` | Units to order (for `restock` actions) |
| `target_zone` | `1` \| `2` \| `3` | `None` | Destination zone (for `place` actions) |

### Observation: `StoremanagerObservation`

| Field | Type | Description |
|---|---|---|
| `inventory` | `List[ProductState]` | Full state of all products (including inactive) |
| `current_step` | int | Current step number (0 at reset, increments after each valid step) |
| `max_steps` | int | Total steps in this episode |
| `steps_remaining` | int | Steps left in the episode |
| `cumulative_profit` | float | Total profit accumulated so far |
| `last_step_profit` | float | Net profit from the most recent step |
| `last_expired_products` | `List[str]` | Names of products that expired this step |
| `zone_capacity` | `Dict[int, int]` | Max products per zone: `{1: 2, 2: 4, 3: 99}` |
| `zone_occupancy` | `Dict[int, int]` | Active products currently in each zone |
| `zone_multipliers` | `Dict[int, float]` | Pick-prob multiplier per zone: `{1: 1.5, 2: 1.0, 3: 0.6}` |
| `holding_cost_this_step` | float | Total holding cost charged this step |
| `placement_cost_this_step` | float | `3.0` if a `place` action was taken, else `0.0` |
| `restock_cost_this_step` | float | Ordering cost if a `restock` was placed, else `0.0` |
| `task_name` | str | Active task: `easy` / `medium` / `hard` / `expert` / `custom` |
| `error` | str \| None | Error message if the action was invalid (step does **not** advance) |
| `metadata` | dict | Per-step breakdown: `step_revenue`, `step_cost`, `expiry_penalty`, `holding_cost`, `placement_cost`, `restock_cost`, `step_profit` |

### ProductState Fields (inside `inventory`)

| Field | Description |
|---|---|
| `product_id` | Unique integer ID |
| `name` | Product name (e.g. `"Milk"`) |
| `cost_price` | Per-unit cost to the store |
| `selling_price` | Current selling price (decreases each time a discount is applied) |
| `base_selling_price` | Original price — never modified, reference for total discount depth |
| `quantity` | Units currently in stock |
| `expiry_step` | Episode step at which this product expires |
| `zone` | Current shelf zone: `1` (premium), `2` (standard), `3` (back) |
| `effective_pick_prob` | Normalized pick probability this step (sums to 1.0 across all active products) |
| `current_discount_pct` | Discount tier applied **this step only** (0 if no discount action taken) |
| `holding_cost_rate` | Holding cost per unit per step as a fraction of `cost_price` |
| `restock_cost_per_unit` | Cost to order one additional unit |
| `restock_lead_time` | Steps until a placed order is delivered |
| `pending_restock_quantity` | Units currently on order (not yet delivered) |
| `pending_restock_arrives_at_step` | Step number when the pending order will arrive |
| `is_active` | `False` if the product is sold out or expired |

---

## Error Handling

When an action is invalid, the step does **not** advance. The returned observation contains an `error` string and reflects the pre-action state. Common error cases:

| Scenario | Error |
|---|---|
| Discount on inactive / sold-out product | `"Product '...' is no longer active"` |
| Restock with `quantity ≤ 0` | `"restock_quantity must be a positive integer"` |
| Restock when pending order exists | `"already has a pending restock order"` |
| Place to same zone product is already in | `"is already in zone N"` |
| Place to zone at capacity | `"Zone N is at capacity"` |
| Invalid `product_id` | `"Invalid product_id"` |

Always check `obs.get("error")` before processing the observation.

---

## Configuration

Control the server via environment variables:

| Variable | Default | Description |
|---|---|---|
| `STORE_TASK` | _(none)_ | Pre-load a task config at startup: `easy`, `medium`, `hard`, or `expert` |
| `STORE_NUM_PRODUCTS` | 8 | Number of products (overridden by `STORE_TASK`) |
| `STORE_MAX_STEPS` | 30 | Episode length in steps (overridden by `STORE_TASK`) |
| `STORE_NUM_CUSTOMERS` | 10 | Customers simulated per step (overridden by `STORE_TASK`) |
| `STORE_MAX_CONCURRENT_ENVS` | 4 | Max parallel WebSocket sessions |

**Task switching at runtime:** A single server instance can serve all four tasks. Pass `"task": "<name>"` in the WebSocket `reset` message and the environment reconfigures immediately without a server restart.

---

## Deploying to Hugging Face Spaces

```bash

openenv push

openenv push --repo-id my-org/store-manager --private

```

After deployment, your space includes:
- **Web Interface** at `/web`
- **API Documentation** at `/docs`
- **WebSocket** at `/ws` — persistent session for low-latency agent interactions

---

## Project Structure

```

StoreManager/

├── __init__.py                       # Module exports

├── models.py                         # Pydantic schemas: Action, Observation, State

├── client.py                         # StoremanagerEnv WebSocket client

├── tasks.py                          # Task configs (easy/medium/hard/expert) + grader

├── inference.py                      # LLM agent runner with structured logs

├── test_ws.py                        # WebSocket integration test suite (7 test groups)

├── openenv.yaml                      # OpenEnv manifest

├── pyproject.toml                    # Dependencies

└── server/

    ├── StoreManager_environment.py   # Core RL environment logic

    ├── app.py                        # FastAPI server (HTTP + WebSocket)

    └── Dockerfile                    # Container image

```