Spaces:
Sleeping
Sleeping
Commit History
-
Set HF_HOME and HF_DATASETS_CACHE for writable caches on Space
Avoid PermissionError on /.cache when HOME is / or empty; align subprocess env. Made-with: Cursor2954c1d -
Set TORCHINDUCTOR cache dirs before torch import (HF Space no passwd uids)
Avoid getpass.getuser()/getpwuid in rootless images by forcing writable inductor and Triton cache paths under /data/outputs or TMPDIR. Made-with: Cursor20b13c9 -
Work around torch mega-cache double registration on Space imports
- Patch CacheArtifactFactory.register to be idempotent before trl/transformers - Set TORCH_COMPILE_DISABLE for training (child env + train_grpo default) Made-with: Cursor6f0a93d -
Enhance TorchGenerator to support optional subfolder for model loading
8c33b70
-
Post-training push to Hugging Face Hub: --hub-model-repo and pilot query params
Made-with: Cursorebcefc4 -
Document HF token via Space secrets, ENV.example, and operational curl for pilot and push_to_hub
- Accept HUGGINGFACE_HUB_TOKEN in server push_to_hub (no token in repo) - SPACE_RUNBOOK: train with best_loss_dir, push best_by_loss to Hub, download archive - ENV.example placeholder for local use Made-with: Cursor1737549 -
Add 8-bit Adam + gradient checkpointing for bf16 training to fit 1.7B on L4
Qwen3-1.7B OOMed on L4 (22GB) during optimizer.step() because fp32 Adam state alone is ~13.6GB. When --bf16 is passed (i.e. we're training a bigger model), switch to paged_adamw_8bit (drops optimizer state ~4x) and enable gradient_checkpointing (drops activation peak). bitsandbytes and accelerate added to requirements for the 8-bit optimizer backend. Made-with: Cursora5ea00e -
Add --bf16 flag and expose it via /train/pilot for bigger models
Path A switches to Qwen3-1.7B which doesn't fit on L4 in fp32. Add a --bf16 flag to the trainer and a matching bf16 query param on the pilot endpoint so we can enable bf16 without hardcoding it per run. Made-with: Cursor6addc33 -
Force Qwen3 chat template to disable thinking mode via tokenizer patch
Diagnosed the 100% clipped_ratio and entropy=0.07 collapse: Qwen3's chat template was auto-inserting <think>...</think> prefixes, and the 0.6B model was burning its entire 512-token completion budget on reasoning text without ever emitting a tool call. The GRPOConfig flag chat_template_kwargs={'enable_thinking': False} was silently dropped by the installed TRL version. Fix: load tokenizer with AutoTokenizer, wrap its apply_chat_template so every invocation passes enable_thinking=False, then hand the patched tokenizer to GRPOTrainer via processing_class (falling back to the older tokenizer kwarg when needed). A startup probe logs whether <think> tokens still appear in the generation prompt. Made-with: Cursor95fada9 -
Fix reward_func reading wrong env layer for tool_calls counters
ReleaseOpsGRPOEnv holds tool_calls_made/tool_calls_failed on the wrapper itself, but reward_func was probing inner.metrics (the underlying ReleaseOpsToolEnv) which never carries those keys. Result: every rollout scored as silent (-3.0) even when TRL reported tool_call_frequency ~5. _resolve_tool_call_counts now walks both the wrapper and its .env child, checking both the merged metrics dict and direct attributes. This fixes the silent-only bug observed in pilot at step 5-15 where tool_call_freq was 4.9 but reward stayed pinned at -3.0. Made-with: Cursor156f912 -
Nuclear reward shaping: silent rollouts get flat -3, active rollouts floor at +1
Prior shaping let a single disastrous exploration rollout (env.reward ~ -2.5) drag group mean low enough that silence had positive group-relative advantage. GRPO then collapsed to pure silence in one gradient step. New design: silent rollouts are a flat -3.0 sentinel. Active rollouts clamp env.reward to [-1.0, 1.5] and add a +2.0 participation bonus, so every active rollout is at least +1.0. This makes the invariant min(active) >= +1.0 > -3.0 = silent hold regardless of group composition. GRPO cannot reinforce silence. Made-with: Cursor31334fa -
Crush-signal reward shaping to force exploration over silence
Previous run: failure rate dropped 95% -> 13% (model learned valid JSON), but silence (-0.80) still beat exploration (~-0.30). Gap too small to overcome group- level variance, so policy still converged to silent after step 10. New shaping makes exploration a HUGE margin winner: NON_TERMINATION_PENALTY = -0.2 SILENT_ROLLOUT_PENALTY = -1.2 (was -0.6, catastrophic now) TERMINAL_ENGAGEMENT_BONUS = +0.3 (was +0.2) PER_CALL_EXPLORATION_BONUS = +0.15 (was +0.05, 3x) MAX_EXPLORATION_BONUS = +0.6 (was +0.3, doubled cap) SUCCESSFUL_CALL_BONUS = +0.10 (new: stacks with per-call) NON_TERMINAL_FLOOR = -0.6 (unchanged) Landscape ordering (silent is catastrophic): safe_ship + 5 successful calls ~ +1.90 exploration (3 successful) ~ +0.25 (positive!) missed_deadline via hold ~ 0.00 unsafe_ship ~ -0.35 silent ~ -1.40 Any rollout with 2+ successful calls bea…101b660 -
Fix reward collapse: floor exploration loss, bonus per-call and per-terminal
Observed collapse with -0.25 invalid_action penalty on a noisy 0.6B policy: at step 5 the model made 2.2 tool calls per rollout with 95% failure rate, accumulating -1.2 reward. Silence (-0.9) then beat exploration, so gradient correctly pushed policy toward doing nothing. Loss hit 0 at step 10, dead run. New layered shaping: NON_TERMINAL_FLOOR = -0.6 (cap exploration punishment) NON_TERMINATION_PENALTY = -0.2 (softer since floor already caps) SILENT_ROLLOUT_PENALTY = -0.6 (make silence strictly worst) TERMINAL_ENGAGEMENT_BONUS = +0.2 (any terminal > any non-terminal) PER_CALL_EXPLORATION_BONUS = +0.05 each, capped at +0.3 Resulting landscape (best to worst): safe_ship ~ +1.35 missed_deadline ~ -0.30 exploration ~ -0.65 <-- beats silence unsafe_ship ~ -0.70 silence ~ -0.80 <-- strict worst Also add /train/kill endpoint so bad runs can be stopped without factory-reboot. Made-with: Cursor7078c21 -
Live training visualization + aggressive reward shaping to prevent 'do nothing' collapse
Server: - Add /train/plot (auto-refreshing HTML) and /train/plot.png (matplotlib curves) - Add /train/live JSON streaming of reward/loss/tool metrics per log step - Add /outputs/ls, /outputs/file, /outputs/archive download endpoints - Add /train/push_to_hub for durable artifact persistence to HF Hub - /train/pilot now takes max_completion_length (default 512) and logging_steps (default 5) Training: - ReleaseOpsGRPOEnv tracks tool_calls_made and tool_calls_failed per rollout - reward_func adds layered shaping: NON_TERMINATION_PENALTY = -0.5 (break clipping collapse) SILENT_ROLLOUT_PENALTY = -0.4 (make 'do nothing' the worst non-terminal option) FORMAT_BONUS_ON_ACTION = +0.05 (keep exploration alive) Resulting ordering pushes policy toward any terminal action before correctness. - LiveLogHistoryCallback persists log_history on every on_log event so /train/live and /train/plot show real-time data. Dataset: - System prompt rewritten from 'Use tools sparingly' (discour…4dde8b9 -
Add compact training summary endpoint
cbf9895 -
Use Qwen3 model for TRL OpenEnv tool parsing
cb4128a -
Adapt GRPO config to installed TRL API
9aacd47 -
Deploy ReleaseOps Arena GPU Space
9aa9d87