Spaces:
Sleeping
Sleeping
Commit History
-
Remove training notebook for ChargebackOps Merchant Agent, consolidating code and documentation for outcome-based RL on Qwen2.5-3B.
73ff1b0
-
fix(notebook): rewrite diag-code as self-contained loader
The old diag cell called make_text_policy() which the v5 sequential eval cell renamed to make_policy() and changed signature (now returns cleanup_fn not the model handle). Old diag broke with NameError. Fix: diag cell is now self-contained — loads its own fresh base + adapter, runs the 3-task diagnostic, frees. Doesn't depend on any intermediate state from the eval cell. Adds ~3 min for the base load but eliminates the cross-cell coupling that broke between v4 and v5.0421766 -
fix(eval): sequential per-checkpoint base load + product-grade docs
Eval cell: - Replace concurrent eval_base + sft_merged_base pattern with sequential per-checkpoint pattern. Each iteration loads ONE fresh base + adapter, evaluates, then fully frees. Avoids accelerate auto-offloading layers to CPU when two 3B fp16 bases attempt to sit on a 15 GB T4 (which caused 'Expected all tensors to be on the same device' in PEFT 0.14 forward pass when GRPO adapter on GPU met SFT base layer on CPU). - Slower (3x base reloads vs 2) but reliable. Each checkpoint eval costs ~3 min model load + ~10-15 min episode rollout = ~15-18 min. Documentation: - README.md rewritten as a product-grade introduction. Leads with the cost-asymmetric multi-round adjudication primitive, generalizes beyond chargebacks, links to all supporting docs. - docs/RESULTS.md rewritten with current per-checkpoint per-family numbers, scripted policy sweep, marathon results, per-dimension rubric attribution, diagnostic rollout. - docs/METHOD.md NEW. Documents the post-SFT GRPO …bb2cdb9 -
perf(training): v5 - shorter SFT, longer GRPO, hard/nightmare oversample
v4 results showed: - SFT collapsed to mean_token_acc=0.96 even at 300 steps -> entropy 0.001 - GRPO improved nightmare 0.55 -> 0.69 but regressed easy 0.92 -> 0.61 - Trained overall 0.73 vs SFT 0.75 vs heuristic 0.81 v5 changes (compounding): - SFT_MAX_STEPS 300 -> 150: stops at loss ~0.20, mean_acc ~0.88, entropy stays higher so post-SFT policy has room to explore. Fixes the SFT collapse problem at the source instead of working around it. - GRPO max_steps 60 -> 200: 3.3x more training steps. v4 showed gradient signal active on ~40% of steps; more steps = more opportunities for policy refinement. - GRPO learning_rate 2e-5 -> 3e-5: 50% bigger updates per step, capitalizes on extra steps before warmup decay. - GRPO data: hard + nightmare difficulties appear 2x in curriculum. Concentrates training where exploration helps (per v4 per-family results) and where the easy-case argmax-lock matters less. - GRPO save_steps 60 -> 80: 2 intermediate checkpoints + final, richer eval…d66354e -
fix(eval): pass offload_folder to from_pretrained on T4 OOM path
Two 3B fp16 bases concurrent in 15 GB T4 = ~12.5 GB peak which leaves no headroom for KV cache + accelerate overhead. accelerate auto-decides some layers belong on CPU/disk; peft 0.14 then crashes: ValueError: We need an offload_dir to dispatch this model according to this device_map, the following submodules need to be offloaded... Fix: pass offload_folder='/content/offload' to every from_pretrained call (eval_base, tmp_base, PeftModel.from_pretrained for SFT and per checkpoint adapter loads). accelerate now has somewhere to spill if device_map auto chose CPU/disk. Slower per-token if offload triggers but doesn't crash.c86ad46 -
fix(grpo): revert num_generations to 8 (TRL gen_batch divisibility)
TRL 0.21 GRPOConfig requires generation_batch_size (= batch x grad_accum = 1 x 8 = 8) to be divisible by num_generations. 8 / 6 = ValueError at config init. The max_steps=60 cut alone (vs old 120) already halves GRPO runtime to ~12min. Cutting num_generations was the wrong knob - max_steps does the heavy lifting and num_generations=8 keeps within-group variance odds healthy (the v1 collapse fix).2dce087 -
perf(notebook): shrink SFT+GRPO budget for Colab free T4
Previous run took ~3hr (SFT 88min + GRPO 24min + eval 30-60min) and exhausted Colab free quota. Cuts that preserve learning quality: SFT: - SFT_TARGET_ROWS 10000 -> 4000 (loss hit 0.10 at step 200/800, rest wasted) - SFT_MAX_STEPS 800 -> 300 (stops at loss ~0.20, mean_acc ~0.92) - save_steps 500 -> 300 (only saves final) GRPO: - max_steps 120 -> 60 (real gradient flow visible by step 30) - num_generations 8 -> 6 (still > 4 for variance, 25% per-step speedup) - save_steps 40 -> 60 (only saves final, eval iterates fewer ckpts) - save_total_limit 3 -> 2 New budget: ~65min total vs ~180min before. Bonus: stopping SFT at loss 0.20 instead of 0.10 leaves entropy higher -> GRPO has more room to learn (fixes the v1 collapse problem at the source instead of working around it with widened sampling).451f087
-
fix(eval): preload bases, swap adapters, fix peft offload_dir crash
Old eval cell loaded a fresh 6.3GB base on every iteration. The 'del m_ckpt; torch.cuda.empty_cache()' pattern did NOT free GPU because the policy closure captured 'm', which captured 'base'. By the second iteration, T4's 15GB was full. accelerate's auto device_map decided to offload some layers to CPU/disk, then peft 0.14 crashed with 'We need an offload_dir to dispatch this model'. Fix: - Free all training-cell globals (model, merged_base, sft_trainer, grpo_trainer, etc) before any eval load. - Load eval_base ONCE (for base + SFT eval). - Pre-merge SFT into a second base ONCE (for GRPO eval). - Per-checkpoint: attach adapter via PeftModel.from_pretrained with unique adapter_name, run eval, then delete_adapter to release the LoRA tensors. Base stays in VRAM the whole time. - Cleanup function returned alongside policy fn so the loop can release adapter weights without nuking the base.8fcb6cf -
fix(grpo): unblock learning - widen sampling + raise lr + keep dropout
Diagnosis from v1 GRPO logs: - grad_norm=0.0 on ~95% of steps - frac_reward_zero_std=1.0 on ~80% of steps (every group of 4 had identical completions producing identical outcomes -> std=0 -> advantage=0) - entropy=0.001-0.017 (post-SFT policy near-delta on argmax) - SFT trained too well (mean_token_accuracy=0.96), then temp=0.7 + top_p=0.9 + top_k=50 picked the argmax 99% of the time. With num_generations=4 that gave 4 identical actions per prompt and zero learning signal. Fixes (combined): - merge cell: lora_dropout 0.0 -> 0.1 (stochasticity in train()-mode gen) - grpo cell: temperature 0.7 -> 1.3 (break the argmax lock) - grpo cell: top_p 0.9 -> 1.0, top_k 50 -> 0 (no truncation) - grpo cell: num_generations 4 -> 8 (doubles within-group variance) - grpo cell: learning_rate 5e-6 -> 2e-5 (bigger push out of SFT basin) - grpo cell: beta 0.0 -> 0.04 (small KL anchor as safety) - grpo cell: removed the dropout-zeroing loop that nuked the merge cell's carefully-set lora_dropout=0.…e26c2ac
-
fix(notebook): pin accelerate==1.0.1 to keep huggingface-hub at 0.26.x
Latest accelerate (1.13+) hard-requires huggingface-hub 1.x. Without the accelerate pin, STEP 2b's only-if-needed install resolved a newer accelerate that in turn upgraded hub past 0.26.5, breaking peft 0.14 and trl 0.21 imports. Pin accelerate to the last 0.x-hub-compatible version (1.0.1) and pull it into STEP 2a alongside the rest of the training stack.5674079 -
feat(training): outcome-based RLVR reward + clean Colab T4 notebook
Replace heuristic-match (distillation) GRPO reward with terminal $-PnL from the multi-round Issuer + arbitration outcome: - training/outcome_reward.py: compute_outcome_reward() runs the model action at the captured state then rolls the heuristic forward; reward is normalized merchant_net_pnl in [-1, +1]. compute_format_reward() provides dense parse-validity shaping for early training. - tests/test_outcome_reward.py: 6 tests pinning bounds, parse-fail, default state_step, length mismatch, format reward, and reward variance across diverse first-actions. - notebooks/train_merchant_agent.ipynb: full rewrite. - Pin downgrade to transformers 4.51.3 + trl 0.21.0 + peft 0.14.0 + tokenizers 0.21.4 + huggingface-hub 0.26.5 (combo tested against Colab + Kaggle pre-install drift, with hard asserts). - Strip ChoiceConstrainedGRPOTrainer subclass, LogitsProcessor monkey patch, A/B/C/D candidate-choice prompts, choice-SFT warmup. Vanilla GRPOTrainer with reward_funcs=[o…1f49d52 -
Implement code changes to enhance functionality and improve performance
30a9e6e -
feat: Implement wait_for_updates action for handling delayed cases and evidence
- Added support for the "wait_for_updates" action in baseline_runner and benchmark_runner to manage scenarios with no open cases. - Enhanced the ChargebackOpsEnvironment to handle pending issuer reviews and delayed evidence, allowing agents to wait for updates when necessary. - Introduced a new marathon task scenario with wave arrivals, simulating long-horizon backlog management. - Updated tests to validate the behavior of the new wait_for_updates action and its integration with delayed evidence and issuer reviews.2dedffd
-
feat: enhance completion parsing to handle truncated JSON and `<think>` blocks
71f1fe0 -
feat: add per-family evaluation and plotting for training curves
a79d430
-
feat: Add training curve evaluation and plotting utilities with unit tests
8fe3b35 -
Add training notebook and benchmark runner for ChargebackOps
- Introduced `train_merchant_agent.ipynb` for end-to-end GRPO training of the merchant-side chargeback agent, including environment setup, model loading, training prompt dataset creation, and evaluation. - Created `benchmark_runner.py` to implement scripted policies for benchmarking against the trained agent, including heuristic, escalate_all, concede_all, and naive policies. - Updated `issuer_model.py` to refine rationale messages for issuer decisions. - Added unit tests for the benchmark runner and training adapter to ensure policy validity and reward computation accuracy. - Implemented training helpers in `training/__init__.py`, `env_adapter.py`, and `reward_adapter.py` to facilitate interaction with the GRPO trainer.bd00c06
