Spaces:
Sleeping
Sleeping
Commit History
-
Refactor grader to use openenv.core.rubrics.WeightedSum + Rubric subclasses
Closes the acknowledged conformance gap. The functional behavior is preserved exactly (52/52 tests pass, including 2 new tests verifying the grader literally uses WeightedSum with 6 named child rubrics). Architecture: server/rubrics.py — 6 Rubric subclasses, one per scored axis: CrashFreeRubric, ProgressRubric, ConnectionRubric, AdaptationRubric, EfficiencyRubric, BeliefAccuracyRubric Each holds a reference to the env in __init__; forward(action, obs) ignores the per-step args (RFC 004 pattern for trajectory-summary scoring) and reads aggregated env state. make_grade_rubric(env) returns a WeightedSum composing all 6 with weights summing to 1.0 (0.15 + 0.20 + 0.10 + 0.25 + 0.10 + 0.20). RhythmEnvironment._grade_episode now lazy-builds and delegates to the WeightedSum on done=True. Also updated: - server/rhythm_environment.py: cached _grade_rubric field on the env - tests/test_rhythm_env.py: 2 tests verifying WeightedSum is used - docs/iterations.md: rep…f0ca22d -
Acknowledge OpenEnv Rubric system conformance gap
We use a custom multi-component grader in _grade_episode that's functionally equivalent to openenv.core.rubrics.WeightedSum (composable weighted scoring, same independent components, same explicit weights) but doesn't use the literal Rubric class. The Rubric API is per-step (action, observation), while our grader reads aggregated episode-end state — a clean refactor would use TrajectoryRubric. Decision: don't refactor in the last 30 min before deadline. Risk of breaking the live HF Space (which IS the submission) outweighs the literal-conformance benefit. Documented as a v2 cleanup task in the code docstring + iterations.md.dc5658d -
Algorithm Distillation: grader v2 with belief_accuracy + SFT pipeline
Grader changes: - Add belief_accuracy term (weight 0.20) to _grade_episode - env.record_belief() for callers to register the agent's emitted belief - Redistribute weights: crash 0.20→0.15, progress 0.25→0.20, conn 0.15→0.10, adapt 0.30→0.25 - Heuristic + random baselines still emit no belief, score 0 on this component - Validates: gpt-5.4 teacher beats heuristic 0.617 vs 0.449 on 30/30 episodes Distillation pipeline: - scripts/generate_teacher_trajectories.py: Azure OpenAI teacher rollouts - scripts/reeval_teacher_trajectories.py: offline re-score under v2 grader - scripts/upload_teacher_data.py: push trajectories to HF Hub dataset - training/sft_prime.py: TRL SFTTrainer fine-tune of Qwen 2.5-3B on teacher data - scripts/sft_on_hf.py: HF Jobs orchestrator for SFT prime stage Parser + prompt: - extract_action_and_belief now handles both single-line and CoT-prefixed formats (takes LAST <digit> <digit> <digit> <ACTION> match) - SYSTEM_PROMPT in dataset.py unified with teacher's CoT-as…ece0bbe -
iter4: fix the 'constant belief = free reward' bug + 6 other deep issues
Investigation prompted by external code review. Validated 5/5 critical findings against actual code; applied fixes for the 7 most impactful. [1] Anomaly signals NOW REACH THE LLM (Issue 1, was CRITICAL) Env was computing per-meter anomalies (actual_delta - expected_under_neutral) and stashing them in reward_breakdown — but format_observation_prompt only read meters and step_history. StepRecord didn't include anomalies. So the cleanest profile-inference signal in the env was COMPUTED BUT NEVER SHOWN to the agent. The agent has been doing meta-learning blindfolded. Fix: extend StepRecord with 5 anomaly fields, populate in env.step, surface in prompt and inference.py. The agent now sees how each person's response deviates from the average baseline — direct profile fingerprint. [2] Belief baseline subtraction (Issue 4, was CRITICAL — biggest impact) OLD: reward = (1 - mae) - 0.5 Constant '5 5 5' emission scored +0.336 raw per step (× 3.0 weight = …bb2a9c7 -
iter3: align reward with grader + belief-first format + exploration shaping
ITER 2 FAILURE: agent escaped single-action collapse only to find a 2-cycle loop (MEDITATE+EXERCISE alternation). final_score 0.22, worse than random. Belief learning (+0.36) works but doesn't influence action choice — belief was emitted as a string AFTER the action, so action couldn't condition on it. Five runtime-level fixes for iter 3: [1] Per-step reward grader-alignment (server/rhythm_environment.py:_compute_reward) Add profile-INDEPENDENT bias: +0.5*progress_delta + 0.4*connection_delta. Profile-weighted reward still drives belief inference (varies by profile), but agent now ALWAYS gets penalized for ignoring progress and connection. Closes the proxy/goal misalignment that let M+E alternation win. [2] Belief-first output format (training/reward_functions.py, dataset.py, inference.py) Was: 'ACTION_NAME S M W' (belief was post-hoc afterthought, didn't condition action) Now: 'S M W ACTION_NAME' (causal LM generates belief tokens FIRST, then action cond…64d24b3
-
iter2: fix mode collapse + 3 deeper bugs from code review
ITER 1 FAILURE: mode collapse to 'EXERCISE 5 5 5' at step 100. Root cause: format_valid (+1.0) and action_legal (+0.5) saturated for any valid output, contributing zero to GRPO advantage. Once 4/4 completions per prompt were identical, reward_std=0 and the policy froze. 7 fixes applied (4 from initial diagnosis + 3 from deep review subagent): [1] sampling temperature 1.0 -> 1.5 Forces diverse rollouts per prompt, breaks mode collapse mechanism [2] reward_weights [0.3, 0.3, 1.0, 1.0] -> [0.05, 0.05, 1.5, 3.0] Aggressively suppress saturated layers, amplify variable signals. Belief weight bumped to 3.0 (CRITICAL-1) so emitting belief is clearly rewarded vs the no-belief penalty. [3] action_legal: drop +0.5 for valid, return 0 instead Removes another constant-reward source. Layer is now pure penalty for malformed outputs. [4] explicit repetition penalty in env_reward (-0.3 if action seen 3+ in row) Direct training signal against mode collapse. Env already …e21a960 -
env: meta-RL refactor (continuous profiles, action+belief, adaptation grader)
Transforms rhythm_env from a 3-profile single-task RL benchmark into a meta-RL environment where the agent must infer the user's hidden personality from observations alone. Original 500-step training regressed vs heuristic on all 3 profiles; this refactor addresses the four root causes: 1. server/rhythm_environment.py - Continuous profile sampling: sample_profile(seed) draws 14 personality parameters from bounded distributions per episode (memorization impossible, agent must learn the *skill* of inference) - Discrete 3-profile mode preserved via profile_mode='discrete' or profile=<name> for legacy comparison - profile_to_belief_vector(): reduces a profile to 3-dim ground truth [social, morning, work] for belief-accuracy reward - get_belief_target() / get_profile_hint(): privileged accessors for training-side reward computation and curriculum hints - Grader rewrite: dropped meter_balance (was rewarding random behavior), added adaptation_scor…ecbe0d8 -
env: enrich observation with history, anomalies, and discovery bonus
Five environment-level changes to make personality discovery learnable: 1. step_history (last 7 steps) added to RhythmObservation — agent now has the raw action/reward/delta trajectory needed to detect profile anomalies across steps, not just the current snapshot 2. Per-meter anomaly signals in reward_breakdown — each step computes actual_delta minus expected_delta (neutral-profile baseline after time-of-day + vitality factor), giving the agent a direct fingerprint of the hidden modifier (e.g. +0.06 vitality_anomaly on DEEP_WORK = workaholic) 3. First-class delta fields on RhythmObservation (vitality_delta, etc.) and last_action — no longer buried in reward_breakdown dict 4. Discovery bonus (15%) added to _grade_episode — rewards profile-adapted strategy in steps 14–27 (second half of week); introvert avoids social, extrovert embraces it, workaholic front-loads work. Without this, the grader rewarded generic meter management and ignored personality infere…0a15ab5
-
Rebuild as Life Simulator: 5 meters, 3 hidden profiles, GRPO training pipeline
server/rhythm_environment.py: full Life Simulator (28 steps, 5 meters, 10 actions, 3 hidden personality profiles with distinct reward weights and action modifiers) models.py: add reward/done fields to RhythmObservation training/: complete GRPO pipeline (dataset, train, reward_functions, inference_eval, Colab notebook); env_reward uses real episode replay via dataset columns ui/app.py: full rewrite for Life Simulator (5-meter display, 10 actions, profile inspector) blog_post.md: mini-blog for hackathon submission docs/round2/: entity_definitions and environment_design updated to Life Simulator spec Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>cc6473a
-
Fix bugs, add tests, and improve code quality
Akhil Soni- Fix START_TASK/SWITCH_TASK semantic distinction (was identical code) - Fix progress reward lost on task completion step (worked_on_task_id) - Fix grader weights summing to 0.95 (now 1.0) - Fix grader efficiency giving idle agents perfect score - Fix grader deadline scoring (now importance-weighted) - Fix reward clamp [-1,1] truncating completion signal (now [-2,2]) - Add auto-clear current_task_id on task completion - Add early termination when all tasks complete - Add custom task mode (task="custom") - Add 27 tests covering reset, step, grader, and edge cases - Add Dockerfile to project root for validation script - Add BSD copyright headers to all files - Remove dead code, unused imports, and unused dependency - Update README with accurate baseline scores and documentationc07f15e -
Add custom task input support and update URLs to HF Space
Akhil Soni- reset() now accepts task="custom" with user-defined tasks, meetings, and energy - Validates custom input (1-10 tasks, bounds clamping on all params) - Existing easy/medium/hard scenarios unchanged - Default URL updated from localhost to HF Spacee74ff96 -
Initial commit: RhythmEnv daily planning RL environment
Akhil SoniA deterministic RL environment simulating daily planning and scheduling under energy, stress, deadline, and importance constraints. - 3 graded tasks (easy/medium/hard) with real-world scenarios - Multi-component reward function with partial progress signals - Baseline inference script with heuristic + LLM agent - OpenEnv spec compliant, Docker ready025774a