kernel-env / train /README.md
Shreyansh327's picture
Publish OpenEnv kernel environment and bounded GRPO proof artifacts
e313085 verified
|
Raw History Blame Contribute Delete
9.03 kB

train/ — baseline eval + GRPO for kernel_env

File What
baseline_eval.py Frozen model on held-out tasks, multi-turn (env's max_turns). Prints compile %, correct % (best of episode and turn 1), faster-than-ref %, mean/median speedup of correct episodes, stage histograms, suspicious-flag count; writes per-episode JSONL.
grpo_kernel.py TRL 1.16 GRPOTrainer + rollout_func. One dataset row = one KernelBench task id; TRL repeats it --num_generations times (= one GRPO group). Each rollout is a full episode: generate → env.step → feedback turn → … Feedback tokens get env_mask=0 (not trained). Reward = Σ γ^t r_t (--return_mode discounted --gamma 0.9) or best turn (--return_mode best).
test_train.py Mac checks: code extraction, return math, token stitching == chat template, rollout vs a mock server.

Both scripts enumerate task ids from a local KernelBench copy (KERNELBENCH_DIR); the train/eval split is the server's hash split, so it matches. Default --levels 1.

Modal

modal_app.py::job (repo root) runs one command in /root/train on a TRAIN_GPU container, with the env server on the last --eval-gpus GPUs ($KERNEL_ENV_URL) and the command on the rest. Outputs go to /runs (Modal volume kernel-env-runs); HF models are cached in kernel-env-hf-cache. Optional secrets: TRAIN_SECRETS=wandb,huggingface (names of your Modal secrets).

# baseline (fast path): `vllm serve` on GPU 0 with MTP speculative decoding, kernel evals on GPUs 1-3;
# episodes run concurrently, so one episode's generation overlaps another's kernel eval
TRAIN_GPU=H100:4 modal run modal_app.py::job --eval-gpus 3 --cmd 'python baseline_eval.py --env-url $KERNEL_ENV_URL \
  --model Qwen/Qwen3.5-9B --max-new-tokens 8192 --max-model-len 65536 --spec-mtp 2 \
  --vllm-args "--language-model-only --reasoning-parser qwen3" --out /runs/baseline_qwen35_9b.jsonl'

# GRPO, Qwen3.5-9B + LoRA: 4 training GPUs (vLLM colocated), 4 kernel-eval GPUs (sizing below)
TRAIN_GPU=H100:8 modal run modal_app.py::job --eval-gpus 4 --reward roofline --cmd 'accelerate launch --num_processes 4 grpo_kernel.py \
  --model_name_or_path Qwen/Qwen3.5-9B --dtype bfloat16 --bf16 true --env_url $KERNEL_ENV_URL \
  --use_peft --lora_r 64 --lora_alpha 128 --lora_target_modules all-linear \
  --use_vllm --vllm_mode colocate --vllm_gpu_memory_utilization 0.3 \
  --num_generations 8 --per_device_train_batch_size 8 --gradient_accumulation_steps 2 \
  --max_completion_length 4096 --learning_rate 1e-5 --gradient_checkpointing true \
  --eval_strategy steps --eval_steps 25 --num_generations_eval 1 --report_to none --output_dir /runs/grpo_q35_9b'

Eval GPU type = training GPU type here (one container). Speedups are only comparable on the same GPU type.

Sizing (Qwen3.5-9B, P = 9.65 B params incl. vision tower)

Formula Qwen3.5-9B
bf16 weights 2·P 19.3 GB
Full fine-tune state (bf16 weights + grads, fp32 master + Adam m, v) 16·P 154 GB: DDP does not fit on 80 GB; ZeRO-3 needs 154/n GB per GPU plus activations and vLLM
LoRA (frozen bf16 base + adapters ≈ 1–2 % of P with fp32 Adam) 2·P + 16·P_lora ≈ 22 GB per GPU before activations
Colocated vLLM vllm_gpu_memory_utilization × 80 GB 0.3 → 24 GB: 19.3 GB weights + ≈ 5 GB KV cache
Rollouts per optimizer step R per_device_bs × processes × grad_accum 8 × 4 × 2 = 64 (8 groups of 8)
Kernel evals per step ≤ R × max_turns ≤ 256
Eval-queue time per step evals × eval_s ÷ eval GPUs 256 × ~10 s ÷ 4 ≈ 11 min

eval_s (≈ 10 s here: process start + Triton compile + 5 correctness trials + 2 benchmarks) is an estimate; read the measured p50/p95 under "Where the time went" in report.py output, and size --eval-gpus so the eval queue does not dominate the step. Full fine-tuning instead of LoRA needs ZeRO-3 (an accelerate DeepSpeed config) on ≥ 4 training GPUs with vLLM in server mode on its own GPU.

GPU box

Keep eval GPUs and training GPUs disjoint, or timing noise leaks into the reward. Example for 8 GPUs: GPUs 0–1 run the env (one kernel eval per GPU at a time), 2–7 train.

# 1. env server on eval GPUs only
docker build -t kernel_env:latest -f kernel_env/server/Dockerfile kernel_env
docker run -d --gpus '"device=0,1"' -p 8000:8000 \
  -e KERNEL_ENV_GPUS=0,1 -e KERNEL_ENV_MAX_SESSIONS=256 kernel_env:latest

Inside the container the two leased devices show up as 0,1, hence KERNEL_ENV_GPUS=0,1.

# 2. trainer env (separate venv; needs vllm for the GPU paths)
cd train && uv venv --python 3.12 && uv pip install trl vllm peft datasets -e ../kernel_env
git clone https://github.com/ScalingIntelligence/KernelBench /opt/KernelBench
export KERNELBENCH_DIR=/opt/KernelBench/KernelBench

# 3. baseline on held-out split (vLLM on a training GPU, not an eval GPU)
CUDA_VISIBLE_DEVICES=2 python baseline_eval.py --env-url http://localhost:8000 \
  --model Qwen/Qwen2.5-Coder-7B-Instruct --backend vllm --out baseline_7b.jsonl

# 4. GRPO: vLLM server on GPU 2, training on GPUs 3-7
CUDA_VISIBLE_DEVICES=2 trl vllm-serve --model Qwen/Qwen2.5-Coder-7B-Instruct &
CUDA_VISIBLE_DEVICES=3,4,5,6,7 accelerate launch --num_processes 5 grpo_kernel.py \
  --model_name_or_path Qwen/Qwen2.5-Coder-7B-Instruct --dtype bfloat16 --bf16 true \
  --env_url http://localhost:8000 \
  --use_vllm --vllm_mode server \
  --num_generations 8 --per_device_train_batch_size 8 --gradient_accumulation_steps 4 \
  --max_completion_length 4096 --temperature 1.0 --learning_rate 1e-6 \
  --gradient_checkpointing true --return_mode discounted --gamma 0.9 \
  --eval_strategy steps --eval_steps 25 --per_device_eval_batch_size 8 --num_generations_eval 1 \
  --logging_steps 1 --report_to wandb --output_dir runs/grpo_7b

Sizing notes:

  • Concurrent env sessions = rollouts in flight across all processes (per_device_train_batch_size × steps_per_generation × num_processes); keep it ≤ KERNEL_ENV_MAX_SESSIONS.
  • Episode wall time is bound by the eval-GPU pool: N sessions queue for 2 GPUs. --env_timeout_s (default 900) must cover that queueing; add eval GPUs before raising it.
  • --max_completion_length is per turn; an episode completion is up to max_turns × that plus feedback, so memory grows with turns.
  • Single-GPU-trainer alternative: --use_vllm --vllm_mode colocate.

Seeing what happened

Source What Where
baseline_eval.py stdout [turn k] live, gen s, slowest step, stage mix, correct % per turn; one line per finished episode terminal / Modal logs
--out run.jsonl one line per episode, written as it finishes: completions, per-turn stage/feedback/speedup/timings (gen_s, step_s), episode_id, env history (gpu_id, queue_wait_s, eval_s, code_sha) /runs/ on Modal
env server one JSON line per reset/step ("event": "step", same episode_id/code_sha); also appended to $KERNEL_ENV_EVAL_LOG stdout; /runs/evals-<time>.jsonl in Modal jobs
--wandb PROJECT per-turn metrics, summary, episode table (Modal: TRAIN_SECRETS=wandb) W&B
report.py run.jsonl -o run.html stage funnel, turn-by-turn progress, speedup distribution, timing p50/p95, every episode with code, feedback and diff between turns static HTML
modal volume get kernel-env-runs baseline_qwen35_9b.jsonl .
python report.py baseline_qwen35_9b.jsonl -o baseline.html --title "Qwen3.5-9B baseline"
grep '"stage": "timeout"' evals-*.jsonl     # server log: join to episodes on episode_id

Logged metrics

kernel/{compile_pct, correct_pct, correct_turn1_pct, faster_than_ref_pct, mean_speedup, suspicious, turns}, under train and eval_ (held-out, every --eval_steps). Episodes the env flags as suspicious (correct and >50× speedup, from env.state().history) are logged as warnings and appended with their code to <output_dir>/suspicious.jsonl. Read those before trusting a speedup jump.

Mac (no CUDA)

mkdir -p /tmp/kb/level1 && for n in 1 2 3 12 19; do cp ../kernel_env/tests/fixtures/level1/19_ReLU.py /tmp/kb/level1/${n}_ReLU.py; done
(cd ../kernel_env && KERNEL_ENV_EVALUATOR=mock KERNELBENCH_DIR=/tmp/kb .venv/bin/uvicorn server.app:app --port 8765) &
KERNEL_ENV_URL=http://127.0.0.1:8765 .venv/bin/python -m pytest -q test_train.py
KERNELBENCH_DIR=/tmp/kb .venv/bin/python grpo_kernel.py --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct --dtype float32 \
  --env_url http://127.0.0.1:8765 --output_dir /tmp/run --use_cpu --report_to none --save_strategy no \
  --num_generations 2 --per_device_train_batch_size 2 --max_completion_length 96 --max_steps 2 --logging_steps 1 \
  --eval_strategy steps --eval_steps 2 --per_device_eval_batch_size 2 --num_generations_eval 2

Copies of the ReLU fixture under several ids put tasks in both splits (1_, 12_ hash to eval). Mock evaluator: real static checks, every passing submission is "correct, 1.00×" (reward 0.3).