Spaces:
Runtime error
Runtime error
Commit History
-
Fix Phase 2 OpenEnv validation traps: add grader paths to openenv.yaml and safe parameterless defaults
699f953 -
Spec-compliance overhaul: remove difficulty_multiplier, weighted blend scoring, dep_hard fix, [END] format
FIXES: 1. inference.py: Remove score= from [END] lines (not in official spec) 2. inference.py: Scoring changed to 0.60*max + 0.40*mean (from pure average) 3. inference.py: All abort/error paths emit spec-compliant [END] with empty rewards 4. base_grader.py: difficulty_multiplier() REMOVED (uniform caps killed variance) 5. dependency_cases.py: dep_hard min_actions=1, seq=['migrate_api'] (was 2x which forced repetition penalty) 6. app.py /inference: Parse scores from final_scores JSON (not task_id/total_reward which didn't exist) 7. app.py /benchmark: Same weighted blend + no score= in [END] 8. README.md: Fix [END] example, update scoring descriptionf3fd4ef
-
Major grading overhaul: difficulty multiplier, tighter scoring, mastery removal, precision penalties
- base_grader: difficulty_multiplier caps easy/medium/hard at 0.99/0.90/0.80 - base_grader: increased repetition (-0.20), invalid (-0.40), harmful (-0.50) penalties - security_grader: CVSS partial credit ±3.0 -> ±1.5, token coverage uses len(tokens) not len-1 - security_grader: propose_fix floor removed, revise_fix floor 0.20->0.10, regression penalty doubled - dependency_grader: removed 0.15 all-correct bonus, precision-weighted flag scoring - dependency_grader: migrate partial credit 0.6->0.25, order violations 0.20->0.30 - clinical_grader: adjacent risk 0.5->0.25, hallucination penalty in rank_issues - clinical_grader: order_steps violation -0.25->-0.35, extra steps -0.10->-0.20 - router: mastery early-exit REMOVED entirely, done by sequence+max_steps only - security_cases: CVSS ranges tightened, required_sequence enforced for all 3 actions - dependency_cases: completion thresholds lowered, tricky compat constraints added - clinical_cases: required_sequence enforced (medium=2 steps,…72b3e8d
-
fix(benchmark): Hardening multi-agent environment and strict score compliance
- Enforce strictly OpenEnv-compliant scores in (0.01, 0.99) across all paths - Implement smart prompt truncation to fix API errors - Introduce domain check LRU caching - Increase dataset size to 41 cases (added 9 new hard tasks) - Convert to Docker multi-stage build mapped with Gunicorn - Overhaul Web UI for correct dashboard logging6f95f2a
-
Clamp scores strictly to (0.01, 0.99) to pass OpenEnv Phase 2 continuous environment score verification checks
829f543 -
first commit
4ec75cf