Spaces:
Sleeping
Sleeping
Commit History
-
feat(training): outcome-based RLVR reward + clean Colab T4 notebook
Replace heuristic-match (distillation) GRPO reward with terminal $-PnL from the multi-round Issuer + arbitration outcome: - training/outcome_reward.py: compute_outcome_reward() runs the model action at the captured state then rolls the heuristic forward; reward is normalized merchant_net_pnl in [-1, +1]. compute_format_reward() provides dense parse-validity shaping for early training. - tests/test_outcome_reward.py: 6 tests pinning bounds, parse-fail, default state_step, length mismatch, format reward, and reward variance across diverse first-actions. - notebooks/train_merchant_agent.ipynb: full rewrite. - Pin downgrade to transformers 4.51.3 + trl 0.21.0 + peft 0.14.0 + tokenizers 0.21.4 + huggingface-hub 0.26.5 (combo tested against Colab + Kaggle pre-install drift, with hard asserts). - Strip ChoiceConstrainedGRPOTrainer subclass, LogitsProcessor monkey patch, A/B/C/D candidate-choice prompts, choice-SFT warmup. Vanilla GRPOTrainer with reward_funcs=[o…1f49d52
-
feat(training): SFT dataset + stall detection in eval rollout
Two permanent fixes for the failure mode that produced eval=0 across all GRPO checkpoints in the prior Colab run: 1. SFT dataset module (training/sft_dataset.py) — heuristic rollouts captured as (prompt, oracle_completion) pairs for supervised pre-training. Two-phase RLHF (SFT then GRPO) is the proven recipe for teaching a base model the JSON schema and per-state action variety before sparse RL reward kicks in. Pure GRPO from a base model collapses to 'always emit select_case' because the model never sees varied action_types in the prompt-only training data. 2. Stall detection in run_episode_with_text_policy — the dominant failure was a checkpoint emitting select_case at every state. The env silently no-ops the duplicate select_case (returns -0.02 reward, advances step_count), so the rollout burns its entire step budget without flipping done. New code: - Hard-coded predicate _predicted_noop catches the dominant case (select_case when a case is al…02a6a9f
-
fix(training): per-action reward scoring vs heuristic oracle
Prior compute_reward fell back to the scripted heuristic on parse-fail, which trained the model that emitting garbage was optimal: the heuristic played the whole episode and earned ~0.96 reward regardless of model output. Group reward variance collapsed to std~=0.005 across 6 wildly different completions, GRPO advantage went to 0, loss collapsed to 0. This rewrites compute_reward as a pure per-action scorer against the heuristic oracle at the dataset's recorded env state: - parse-fail -> 0.0 - action_type not in available_actions at this state -> 0.1 - valid action_type, different than oracle -> 0.4 - right action_type, wrong target (case_id/system) -> 0.7 - exact match on action_type + targets -> 1.0 Variance across the same 6 diverse completions: std=0.39 (70x lift), distinct values {0.0, 0.1, 0.7, 1.0}. GRPO now has real gradient. Adds build_state_action_dataset() to roll the heuristic and capture (state, oracle_action) pairs, so trainin…243aa68 -
feat: enhance completion parsing to handle truncated JSON and `<think>` blocks
71f1fe0 -
feat: add per-family evaluation and plotting for training curves
a79d430
-
feat: Add training curve evaluation and plotting utilities with unit tests
8fe3b35 -
Add training notebook and benchmark runner for ChargebackOps
- Introduced `train_merchant_agent.ipynb` for end-to-end GRPO training of the merchant-side chargeback agent, including environment setup, model loading, training prompt dataset creation, and evaluation. - Created `benchmark_runner.py` to implement scripted policies for benchmarking against the trained agent, including heuristic, escalate_all, concede_all, and naive policies. - Updated `issuer_model.py` to refine rationale messages for issuer decisions. - Added unit tests for the benchmark runner and training adapter to ensure policy validity and reward computation accuracy. - Implemented training helpers in `training/__init__.py`, `env_adapter.py`, and `reward_adapter.py` to facilitate interaction with the GRPO trainer.bd00c06
