Spaces:
Sleeping
Sleeping
Commit History
-
docs: make train_trl_grpo.py the primary training path
garg-tejasUpdate launcher recommendations and README guidance to standardize on the TRL-native script as the main training entrypoint, while keeping older scripts explicitly marked as legacy alternatives.667c189 -
fix: pin vllm to TRL-supported range in HF jobs
garg-tejasPrevent HF training jobs from installing vLLM 0.19.x, which triggers TRL compatibility warnings and risks runtime issues.8c7b87a -
fix: random-step dataset + trajectory cache + experiment tracking for L40S
garg-tejas- Add server/greedy_policy.py: shared greedy dispatcher with graph travel times - Add server/trajectory_cache.py: parallel cache build, random-step sampling, replay - Refactor train_trl_grpo.py: - build_seed_dataset now samples random steps from greedy trajectories - make_reward_fn replays cache, injects model action, continues greedy - Fixes step-0-only training bug (model now learns all steps) - Add --num-generations, --cache-workers, --steps-per-seed, --cache-dir CLI args - Defaults optimized for L40S: 8 gens, 8 workers, 7 steps/seed, /data cache - Add wandb + TensorBoard experiment tracking - Update server/prompt_utils.py: explicit reward structure in SYSTEM_PROMPT - Update scripts/launch_hf_job.py: - Add --script grpo option - Add --wandb-project, --wandb-entity, --wandb-run-name, --tensorboard - Pass WANDB_API_KEY to job env - Default flavor: l40sx1, default num_generations: 8 - Update train_grpo.py: add HF Hub push, metrics.jsonl logging - Update working_note…6da8b7c -
feat: rich step-by-step episode trace logging
garg-tejasPrint every action + state snapshot so training progress is readable: 📋 Episode 3/10 | seed=42 | diff=hard | calls=[cardiac@node5, fire@node12] Step 1: dispatch:unit0:call2 | calls=2(pending=1) resolved=0 fatal=0 | ... Step 2: wait | calls=2(pending=1) resolved=0 fatal=0 | New call: trauma@node7 ... ✅ Episode 3/10 DONE | seed=42 | steps=45 | reward=0.823 - Adds _format_step_log() to DispatchRGRPOEnv for concise per-step summaries. - Prints episode header with initial calls. - Prints every step (not just every 10) during reward_fn rollout. - Fixes env.reset() to pass seed/difficulty explicitly.aeebcdd -
feat: add L40S GPU support (48GB, best availability for TRL script)
garg-tejasA100 Large is often stuck in queue. L40S (48GB, .80/hr) is the best available alternative for colocated vLLM + training. scripts/launch_hf_job.py: - Add 'l40s' flavor (48GB VRAM, .80/hr, recommended for TRL script). - Update docstring to recommend l40s as primary choice. train_trl_grpo.py: - Dynamic vLLM memory utilization: 0.55 on <70GB GPUs (L40S 48GB), 0.40 on A100 80GB. - GPU check now warns only on <30GB VRAM (L4/A10G), not on L40S. - L40S is treated as vLLM-friendly.3afbcc1 -
feat: add TRL-native GRPO training script + updated launcher
garg-tejastrain_trl_grpo.py: - New training script based on Kube SRE Gym hackathon winner pattern. - Uses TRL GRPOTrainer with vLLM colocation for 5-10× speedup on A100. - Dataset as seed manifest (seed + difficulty + initial prompt). - CurriculumCallback for mid-training difficulty escalation. - reward_fn runs full episode (80 steps) with policy model handle from TRL. - rollout_func with generate_rollout_completions for multi-step vLLM (fallback to HF generate if trl.experimental.openenv unavailable). - Explicit vllm_gpu_memory_utilization=0.4 to avoid OOM on A100 80GB. - gradient_checkpointing=True with use_reentrant=False. - LoRA config, HF Hub push, trajectory logging. scripts/launch_hf_job.py: - Replaces old launcher with support for both 'trl' and 'unsloth' scripts. - Flavor-aware defaults (a100-large → trl, l4x1 → unsloth). - Per-script dependency lists (DEPS_TRL vs DEPS_UNSLOTH). - --before-after flag for pre/post training inference comparison. - Artifact upload to companion HF Datase…837ed68 -
fix: address 3 bugs introduced in previous commits
garg-tejasBug 1 (OOM risk): Restored per-step del block for large tensors in both training scripts. Previously removed entirely; now tensors are freed each step, but orch.cuda.empty_cache() stays outside the 80-step loop. Bug 2 (prompt_lens): measure_completion_lengths.py transformers fallback used inputs.input_ids.shape[1] (scalar padded length) instead of per- sequence lengths. Fixed to inputs.attention_mask.sum(dim=1).tolist() so shorter prompts aren't truncated. Bug 3 (infmin): format_observation rendered false_alarm deadline as 'deadline=infmin'. Special-cased float('inf') to render 'deadline=none', matching the system prompt description.3d96a95 -
perf: fix measure_completion_lengths.py sequential vLLM calls
garg-tejasThe script was calling vLLM one prompt at a time inside a loop, completely defeating vLLM's continuous batching. With 10 episodes x 80 steps = 800 sequential calls, it was as slow as transformers. Changes: - Rewrite measure() -> measure_batched(): batch all active env prompts per step into a single vLLM/generate() call (80 calls total, not 800). - Add --max-new-tokens CLI arg (default 512, was hardcoded 2048). - Default model changed from bnb-4bit to plain Qwen/Qwen3-4B (vLLM doesn't support BNB quantization well). - Add warning when user tries vLLM with a BNB-quantized model ID. - transformers fallback also uses batched generation.283ff0d
-
fix: address critical training-blocking bugs from audit
garg-tejasB1 (broken feature): Remove undefined _try_resume_from_hub and --resume-from-hub flag from train_unsloth_grpo.py and launch_hf_job.py. The function never existed. B3 (crash): Replace model.device with next(model.parameters()).device in train_grpo.py. model.device is None under device_map='auto' (4-bit mode). B2 (mechanics bug): Move heatwave_active decrement from generate_call() to tick(). Previously decremented per-call, so heatwave lasted 20 calls instead of 20 steps. In warmup (3 calls) it never expired. B4 (reproducibility): Fix seed override in grpo_env_wrapper.py. The RNG was replaced AFTER _env.reset(), so the episode was already generated with the old seed. Now seed is applied before reset. B8 (logic error): Disable env internal curriculum when training script manages difficulty externally. Added _use_internal_curriculum flag; training script sets it False when --curriculum is active.1827386 -
perf: use vLLM for fast inference in completion length measurement
garg-tejasscripts/measure_completion_lengths.py now supports --use-vllm for much faster inference (no training needed, so vLLM's PagedAttention and continuous batching give 3-5x speedup over transformers). - Falls back to standard transformers if vLLM not installed - Removed Unsloth dependency (not needed for inference-only) - Clean generate_fn abstraction so switching is seamlessaebbf2e -
fix: address critical code review bugs (fatality, reward, dedup)
garg-tejasHigh severity fixes: - dispatcher_environment.py: fatality flag now checked on BOTH active and resolved calls. Previously resolved-late calls (cleared after deadline) never got the fatality flag, underreporting deaths in ground truth. - dispatcher_environment.py: use total_reward (includes perfect-run bonus) instead of episode_reward for GRPO training signal. Bonus was computed but never applied. Medium severity fixes: - grpo_env_wrapper.py: remove false 'verify is free' docstring. verify() consumes a step via _env.step(). - train_unsloth_grpo.py + train_grpo.py: add gradient clipping (clip_grad_norm_=1.0) before optimizer.step() for training stability. - Extract duplicated SYSTEM_PROMPT, format_observation, build_chat_prompt into server/prompt_utils.py. Updated inference.py, train_grpo.py, train_unsloth_grpo.py, test_prompt.py to import from shared module. Also adds scripts/measure_completion_lengths.py for empirically sizing max_completion_length before training.ee35947 -
chore: remove legacy HF training launch scripts and ignore local env file
garg-tejasRetire older launch/monitor helpers that have been replaced by the newer HF job flow, and stop tracking local .env files to avoid accidental commits.df5be79 -
feat: add completion length measurement script
garg-tejasscripts/measure_completion_lengths.py runs inference with a large max_new_tokens and records actual completion lengths. Prints stats (min, mean, median, P90, P95, P99, max) and suggests an optimal max_completion_length with 20% headroom above P95. Usage: python scripts/measure_completion_lengths.py --episodes 10ff59d0d -
fix: smarter defaults for HF Jobs launcher to fit in 8h timeout
garg-tejas- Default batch_size 8 -> 16 (L4 has 24GB, fits comfortably) - Default max_completion_length 1536 -> 256 (cuts generation time ~3x) - Add --max-completion-length CLI arg to launcher - Pipe through to both training and inference commands - Show max tokens in launch summary7b1297a -
feat: training tracker, HF Jobs launcher, inplace fix, visualization
garg-tejasMajor update bringing together recent work: - server/training_tracker.py: real-time batch reports, CSV logging, 6-panel dashboard plots, moving averages - train_unsloth_grpo.py: integrated tracker, model.train()/eval() toggle - server/unsloth_grpo_utils.py: fix inplace operation with .clone() - inference.py: add --use-unsloth, --trajectory-file for before/after - visualize_env.py: --frames-per-step CLI arg for faster rendering - scripts/launch_hf_job.py: HF Hub Jobs launcher (serverless GPU) - scripts/plot_training.py: plot metrics.json training curves - README.md: updated for HF Hub Jobs workflow - app.py: revert to minimal placeholder (no Gradio training UI) - Dockerfile: revert CMD to sleep infinity - pyproject.toml: remove gradio, add proper deps27d2e57 -
fix: add openenv-core dependency to HF Jobs launcher
garg-tejasThe previous job failed with ModuleNotFoundError: No module named 'openenv' because openenv-core was not listed in the uv run --with dependencies. - scripts/launch_hf_job.py: add --with openenv-core[core] to deps list - Also add jmespath which is required by the environmentc43d43f -
rebrand: Dead Air -> DispatchR across all code, docs, and metadata
garg-tejase687796 -
fix: Dockerfile with user 1000, updated launch script
garg-tejas05926e6 -
chore: remove personal guide docs from repo, keep locally only
garg-tejas3d18ad9 -
feat: add HF Spaces deployment guide, Dockerfile, launcher scripts, and Hub push support
garg-tejas02644f2