File size: 5,342 Bytes
e0256aa
fdc2080
 
 
 
 
 
e0256aa
64b4cae
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dd41132
 
 
64b4cae
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
---
title: Smart Study Session Optimizer
emoji: πŸ“š
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
---
# Smart Study Session Optimizer

## Why I Built This

Okay so honestly this project started from a very relatable place β€” I used to study for 4-5 hours straight
during exam prep, and by the third hour I was basically just staring at my notes without retaining anything.
I knew I was supposed to take breaks but it felt like "wasting time". Turns out I was doing it completely wrong.

I came across some research on the Pomodoro Technique and ultradian rhythms, and I got curious: could you
actually *learn* the optimal study break pattern using reinforcement learning? The environment models a simplified
version of mental energy and focus, and lets an RL agent figure out when to grind and when to rest.

It's not a perfect simulation of human cognition (not even close), but it's a fun approximation and it
passed all the hackathon criteria, so I'm calling it a win.

---

## Project Structure

```
study-optimizer/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ models.py        # Pydantic models (Observation, Action, StepResult)
β”‚   β”œβ”€β”€ environment.py   # Core RL environment logic
β”‚   └── client.py        # HTTP client for the environment API
β”œβ”€β”€ server/
β”‚   └── app.py           # FastAPI server
β”œβ”€β”€ inference.py         # Main script: LLM agent runs an episode
β”œβ”€β”€ Dockerfile
β”œβ”€β”€ requirements.txt
└── openenv.yaml         # OpenEnv spec compliance
```

---

## How It Works

The environment simulates a study session. At each step, the agent sees:
- **Energy** (0–100): how much mental stamina is left
- **Focus** (0–100): current concentration level  
- **Time studied**: cumulative minutes of studying
- **Session elapsed**: total minutes including breaks
- **Consecutive study minutes**: streak without a break (focus drops fast here)
- **Break count**: how many breaks taken
- **Productive minutes**: minutes where focus was above 60

The agent picks one of 3 actions:
- `0` β†’ **Study** (5 min, drains energy & focus)
- `1` β†’ **Short break** (10 min, moderate recovery)
- `2` β†’ **Long break** (20 min, full recovery but costs time)

---

## The 3 Tasks

| Task ID | Goal | Score Formula |
|---------|------|---------------|
| `survive_8_hours` | Keep energy > 0 for 480 min | `clamp(energy_remaining / 100, 0.01, 0.99)` |
| `maximize_productivity` | Max productive minutes in 240 min | `clamp(productive_minutes / 240, 0.01, 0.99)` |
| `smart_break_taker` | Only break when energy < 40% | `clamp(good_breaks / total_breaks, 0.01, 0.99)` |

---

## Quick Start

### 1. Install dependencies

```bash
pip install -r requirements.txt
```

### 2. Run the server

```bash
uvicorn server.app:app --host 0.0.0.0 --port 7860 --reload
```

### 3. Run inference (LLM agent plays the game)

```bash
# Set env vars first
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4o-mini"
export HF_TOKEN="your_hf_token"

python inference.py --task maximize_productivity
```

### 4. Try it manually with curl

```bash
# Start an episode
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "maximize_productivity"}'

# Study for 5 minutes
curl -X POST http://localhost:7860/step \
  -H "Content-Type: application/json" \
  -d '{"action": 0}'

# Health check
curl http://localhost:7860/health
```

---

## Docker

```bash
# Build
docker build -t study-optimizer .

# Run
docker run -p 7860:7860 study-optimizer

# With env vars for inference
docker run -p 7860:7860 \
  -e API_BASE_URL="https://api.openai.com/v1" \
  -e MODEL_NAME="gpt-4o-mini" \
  -e HF_TOKEN="your_token" \
  study-optimizer
```

---

## Environment Variables

| Variable | Description | Required |
|----------|-------------|----------|
| `API_BASE_URL` | Base URL for OpenAI-compatible LLM API | Yes (for inference) |
| `MODEL_NAME` | Model identifier (e.g. `gpt-4o-mini`) | Yes (for inference) |
| `HF_TOKEN` | HuggingFace API token | Yes (for HF Spaces) |
| `PORT` | Server port (default: 7860) | No |

---

## OpenEnv Compliance

This environment follows the OpenEnv spec:

- `POST /reset` β†’ returns initial observation
- `POST /step` β†’ takes action, returns `{observation, reward, done, info}`
- `GET /health` β†’ returns `{"status": "ok"}`
- `GET /state` β†’ returns full env state
- `openenv.yaml` β†’ spec definition file

---

## Reward Design

Reward at each step:
- Studying with high energy + focus: **+1.0**
- Studying with just energy (focus low): **+0.3**
- Studying while exhausted: **-0.5**
- Smart break (energy < 40): **+0.5 to +1.0**  
- Unnecessary break: **-0.3 to -0.6**
- Terminal reward at end varies by task

The reward shaping is intentionally conservative β€” it's meant to guide the agent
without being so prescriptive that it removes the need to learn anything interesting.

---

## Known Limitations / TODOs

- The energy drain model is simplified (doesn't account for circadian rhythms, caffeine, etc.)
- No persistence β€” session state resets on server restart
- Haven't tried training a proper PPO agent yet, only tested with LLM inference
- Should add a WebSocket endpoint for real-time streaming

---

*Built for the Meta PyTorch OpenEnv Hackathon by Scaler School of Technology*