Spaces:
Paused
Paused
Commit History
-
Make self-play training resilient to HF Space restarts
siddeshwar-kagatikarThe L40S full run was getting killed at ~step 84/120 of the generator phase because uvicorn (PID 1) caught a SIGTERM from the Space and the backgrounded training subshell died with the container, with no recovery path. The answerer reward graph was empty because training never reached the answerer phase. scripts/space_start.sh: invert the lifecycle. uvicorn now runs in the background (so HF healthchecks pass) and self-play training runs in the foreground as the script's blocking child. A SIGTERM from the platform now propagates to the trainer instead of cleanly tearing down PID 1, and the EXIT trap stops uvicorn afterwards. src/osint_env/training/self_play.py: _train_grpo_phase now resumes work on every restart. It (1) skips the phase entirely if final_model is already on disk, (2) auto-detects the latest local checkpoint-* and calls trainer.train(resume_from_checkpoint=...) when full state (optimizer.pt + trainer_state.json) is present, (3) falls back to downloading the latest check…2e14f6d
-
Minor config changes
3e893cd -
Update training config, add checkpointing on HF
e44cdee -
Sync current main to Hugging Face Space
siddeshwar-kagatikarMade-with: Cursorfe1f842 -
fix(rewards): never crash GRPO on malformed completions
siddeshwar-kagatikarTrainer crashed with ``ValueError: invalid literal for int() with base 10: 'none'`` because the model emitted ``"none"`` as a string for an orchestrator integer field. A single bad completion takes down the whole RL run, so harden parsing end-to-end: - Add ``_coerce_int`` that gracefully handles ``None``, booleans, floats (including NaN/inf), and arbitrary strings (``"none"``, ``"N/A"``, ``"2 agents"``). Use it for every orchestrator integer. - Wrap ``decode_completion_text`` and ``parse_generated_task_completion`` in defensive ``try/except`` inside ``GeneratorRewardFunction.__call__``; on any parsing/reward exception, fall through to a graded floor reward so GRPO keeps training and the failure is logged via ``[reward_debug][parse_error]`` / ``[reward_debug][reward_error]``. - Add regression tests for both behaviours. Made-with: Cursord814291