Spaces:
Running
Running
Commit History
-
Replace total_cost with cost_per_instance (average cost per instance)
openhands +1: openhands- Updated simple_data_loader.py to read cost_per_instance instead of total_cost - Changed aggregation to calculate average cost instead of sum - Updated leaderboard_transformer.py column mappings - Updated ui_components.py tooltip descriptions - Updated generate_mock_jsonl.py for mock data generation Co-authored-by: openhands <openhands@all-hands.dev>b1f3e49 -
Fix TypeError when summing costs with None values
openhands +1: openhandsFilter out None values before summing costs to prevent TypeError. Co-authored-by: openhands <openhands@all-hands.dev>cdd40ba
-
feat: Update leaderboard calculations and add incomplete entries toggle
openhands +1: openhandsChanges: 1. Rename 'OpenHands Version' to 'SDK Version' 2. Rename 'Overall Score' to 'Average Score' - now divides by 5 regardless of categories completed (missing categories count as 0) 3. Rename 'Overall Cost' to 'Total Cost' - now sums all category costs 4. Add 'Show incomplete entries' toggle (default: hidden) to filter entries that don't have all 5 categories submitted 5. Fix agent grouping to use version+model combination instead of just version (fixes issue where different models with same SDK version were incorrectly merged) 6. Fix model_dump() to use mode='json' for proper enum serialization 7. Track 'categories_completed' count per agent Co-authored-by: openhands <openhands@all-hands.dev>5998027 -
feat: Use pydantic schema models from openhands-index-results for validation
openhands +1: openhands- Keep full repo clone instead of just results/ directory - Import Metadata and ScoreEntry models from validate_schema.py - Validate agent data during loading with graceful fallback - Log validation errors to surface data quality issues This ensures data is validated against the same schema used by the results repo, catching invalid entries early. Co-authored-by: openhands <openhands@all-hands.dev>f0339f3
-
Update fallback category mappings: place SWE-Bench Multimodal under 'Frontend Development' and Swt-Bench under 'Test Generation'.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsb42a4fe -
Move commit0 to 'App Creation' category in fallback mappings.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsb16f7da -
CRITICAL FIX: Add fallback category mappings for data without agenteval.json
openhands +1: openhandsROOT CAUSE: The app was calculating 0.0 scores because: 1. No agenteval.json file exists in the data directory 2. Without it, benchmark_to_categories dict was empty 3. Category aggregation was skipped (line 162-167 in simple_data_loader.py) 4. Overall score calculation received empty list, resulted in None/0.0 SOLUTION: Add hard-coded fallback mappings when agenteval.json is missing: - swe-bench, swe-bench-multimodal, commit0, multi-swe-bench → Bug Fixing - swt-bench → Frontend Development - gaia → Information Gathering This ensures category aggregation works even without config file. Tested locally: ✅ Shows real scores (57.767, 54.692, etc.) Co-authored-by: openhands <openhands@all-hands.dev>b4ac443 -
Add debug logging to track data loading on HuggingFace Space
openhands +1: openhands- Add data integrity check in setup_data.py - Add sample data logging in simple_data_loader.py - This will help identify where the zero scores are coming from Co-authored-by: openhands <openhands@all-hands.dev>044cdf4 -
Fix score calculation to match AstaBench methodology and update categories
openhandsCritical fixes: - Changed overall score calculation from simple average of all benchmarks to macro-average of category averages - This ensures each category contributes equally regardless of benchmark count - Updated Categories Attempted from 4 to 5 categories (Bug Fixing, Frontend Dev, App Creation, Test Gen, Info Gathering) - Updated category list in leaderboard_transformer.py line 281 - Updated tooltip text to reflect 5 categories This matches the AstaBench calculation methodology.e734bf6 -
Fix agent_version display and make Overall Score bold
openhands- Fixed case mismatch in simple_data_loader.py: 'openhands version' -> 'Openhands version' - Fixed case for 'Language model' -> 'Language model' to match transformer expectations - Modified format_score_column to make Overall Score values bold - Overall Cost remains non-bold as requested0e14c25 -
Rename columns: Agent→OpenHands Version, Models Used→Language Model, remove Submitter
openhands +1: openhands- Updated simple_data_loader.py to use new column names - Updated leaderboard_transformer.py with fixed_mappings and scatter plot references - Updated ui_components.py tooltips and column handling - Changed metadata.json: agent_name→agent_version across all mock data - Added 'Bug Fixing' tag to swe-bench benchmarks in scores.json - Removed 'Submitter' column from base_cols and column references Co-authored-by: openhands <openhands@all-hands.dev>376500e -
Cleanup codebase: remove unused code, simplify data loading, and add pre-release notice
openhands +1: openhands- Remove unused github_data_loader.py (71 lines of dead code) - Clean up debug logging in app.py (replaced ~40 print statements with proper logger) - Remove scheduler code and APScheduler dependency from requirements.txt - Simplify simple_data_loader.py by removing old JSONL format fallback - Update setup_data.py to fetch data from results/ directory only - Update mock data to use new agent-centric directory structure - Add pre-release notice to main page intro paragraph Co-authored-by: openhands <openhands@all-hands.dev>855423e -
Simplify leaderboard to open vs closed models only
openhandsRemove agent tooling distinction from the leaderboard: - Simplified openness from 4 categories to 2: 'open' and 'closed' - Removed all 'tool_usage' references from data structures - Updated UI to remove Agent Tooling column and icons - Simplified scatter plots to use only 2 colors (open/closed) - Updated legends to show Model Openness instead of Agent Openness/Tooling - Regenerated mock data without tool_usage field - Added openness normalization to handle legacy values Files modified: - aliases.py: Simplified openness constants, removed tool usage - simple_data_loader.py: Added openness normalization, removed tool_usage loading - leaderboard_transformer.py: Removed Agent Tooling column, simplified color mapping - ui_components.py: Removed tooling icons/tooltips, simplified to openness only - generate_mock_jsonl.py: Removed tool_usage from mock data generationca754bb
-
Update data loader to support agent-centric directory structure
openhands +1: openhands- Add support for new results/YYYYMMDD_model/ directory structure - Each agent directory contains metadata.json and scores.json - Maintain backward compatibility with old JSONL format - Update mock data to use new structure Co-authored-by: openhands <openhands@all-hands.dev>e003f7b -
Update UI: All-Hands-AI color scheme, agent version column names, and OpenHands logo
openhands +1: openhands- Apply All-Hands-AI color scheme (yellow accent, indigo links, beige background) - Change 'Agent' column to 'Agent Version' - Remove 'Submitter' column - Change 'Models Used' to 'Model' - Update mock data with version numbers (1.0.2, 1.0.1, etc.) - Replace logo with OpenHands branding Co-authored-by: openhands <openhands@all-hands.dev>0ee2099 -
Fix category cost calculation - add category-level aggregation
openhands +1: openhands- Updated simple_data_loader.py to build proper category-to-benchmarks mapping from agenteval.json - Added logic to calculate category-level aggregate scores and costs from individual benchmark results - Updated agenteval.json to include category tags (Bug Fixing, App Creation, etc.) - Fixes issue where Bug Fixing cost was showing as 0.0 instead of the average of swe-bench and swe-bench-multimodal costs Co-authored-by: openhands <openhands@all-hands.dev>5e9c3b9 -
Fix null columns by mapping source data to expected column names
openhands- Modified simple_data_loader.py to use lowercase column names that match transformer expectations - Changed agent_name → agent, llm_base → models used, tool_usage → agent tooling, submission_time → date - Fixed leaderboard_transformer.py tag map capitalization to match column names - All 24 columns now populated correctly (no null values) - Resolves JavaScript table rendering erroraa07520 -
Convert to JSONL data format and remove agent-eval dependency
openhands +1: openhands- Replace parquet files with JSONL for better readability and simplicity - Remove agent-eval library dependency to avoid secret requirements - Create simple_data_loader.py as lightweight replacement for LeaderboardViewer - Create submission_utils.py for submission handling functions - Simplify Dockerfile (no SSH keys or secrets needed) - Reduce requirements.txt to essential packages only - Generate mock data for all 6 benchmarks (swe-bench, multi-swe-bench, swe-bench-multimodal, swt-bench, commit0, gaia) - Update aliases.py to define constants locally - Fix imports in submission.py and test files Co-authored-by: openhands <openhands@all-hands.dev>1027cfb