Spaces:
Sleeping
Sleeping
Commit History
-
Show ACP agent results in the leaderboard (#11)
- Show ACP agent results in the leaderboard (bef7adee8f9654453b27b8d6be41b13e49431cbc) - Move alternative agents to a dedicated page (d7034b85ee14f6dc4445cc0379396ca6dd77c19d) - Show harness in scatter-plot hover on Alternative Agents page (f00a617b2ae09d13284067736bcd5479d30f26ed) - Add Alternative Agents page screenshot (55dc87fa37ef618d4a372d48ef29fd887951a7b6) - Add harness logo assets for Alternative Agents composite markers (6860223b90f3ee4aca8a66e95f57e3fc523a3a54) - Stack model + harness logos on Alternative Agents scatter (fb16d57ae9b3b8e401e09be3af5617febb4a0528) - Replace harness badge letters with generic pictograms (8a1c0edd35eff9780aa5ba987941514c278821f3) - Move harness logos to assets/harnesses/ with graceful missing-file fallback (282557d8a4eb676f672d6d36bf7f25ffb522140d) - Add harness logos for Claude Code, Codex CLI, Gemini CLI, OpenHands (e6877f36e0135f17611518da4254aeb3730482a3) Co-authored-by: Simon Rosenberg <simonrosenberg1@users.noreply.huggingface.co>6d3b657
-
Date in summary table is missing (#10)
- Date in summary table is missing (45fc3a4337e324d17f54b9564db1dbc11951ff36)7949205
-
fix: Add Visualization column to main table (not just benchmark tables)
openhands +1: openhandsThe previous commit only added visualization links to individual benchmark detail tables. This commit adds the Visualization column to the main front page table as well. Changes: - Add 'visualization' column to record in simple_data_loader.py - Add 'Visualization' to ending_cols in leaderboard_transformer.py - Add make_visualization_icons() to format the column in ui_components.py - Update column widths and datatypes to include Visualization - Add tooltip description for Visualization column Co-authored-by: openhands <openhands@all-hands.dev>5f628a6 -
feat: Add Visualization column for Laminar eval links
openhands +1: openhandsDisplay eval_visualization_page URLs as clickable 📊 icons in a new Visualization column next to Logs. Changes: - Load eval_visualization_page from score entries in simple_data_loader.py - Store as '{benchmark} visualization' column similar to download URLs - Add 📊 column to benchmark detail tables with clickable links - Only show icon if visualization URL exists, otherwise blank - Add tooltip documentation for the new column Co-authored-by: openhands <openhands@all-hands.dev>dfa8bfc
-
Skip entries with hide_from_leaderboard=True
openhands +1: openhandsFilter out models from the leaderboard when their metadata has hide_from_leaderboard set to true. This allows hiding models that shouldn't be displayed (e.g., due to lack of comparable models in similar size classes). Co-authored-by: openhands <openhands@all-hands.dev>2123552
-
Remove agenteval.json, use hardcoded mappings as single source of truth
openhandsSimplify the data loader by removing the agenteval.json config file and its loading logic. The benchmark-to-category mappings are now defined directly in the code, eliminating redundancy and potential inconsistencies.24ccc42
-
Add Evolution Over Time and Open Model Accuracy by Size visualizations
openhands +1: openhands- Add visualizations.py with two new chart functions: - create_evolution_over_time_chart: Line chart showing model performance evolution over release dates - create_accuracy_by_size_chart: Scatter plot showing accuracy vs parameter count for open-weights models - Update simple_data_loader.py to load new metadata fields (release_date, parameter_count_b, active_parameter_count_b) - Update main_page.py to display the new visualizations below the leaderboard - Update mock data with release dates and add sample open-weights models with parameter counts These visualizations will display data once the openhands-index-results PR with release_date and parameter_count fields is merged. Co-authored-by: openhands <openhands@all-hands.dev>a4b9436 -
Add runtime column and Cost/Performance + Runtime/Performance charts to all pages
openhands +1: openhands- Add runtime tracking to data loader (simple_data_loader.py) - Add format_runtime_column function (leaderboard_transformer.py) - Update _plot_scatter_plotly with plot_type parameter ('cost' or 'runtime') - Update chart titles to 'OpenHands Index XXX Cost/Performance' and 'OpenHands Index XXX Runtime/Performance' - Add Runtime column to main leaderboard and benchmark details tables - Display both cost/performance and runtime/performance scatter plots on all pages Co-authored-by: openhands <openhands@all-hands.dev>2854ddd
-
Move Download column to benchmark-specific tables only
openhands +1: openhands- Download column now appears on benchmark-specific leaderboard tables (e.g., SWE-Bench, GAIA) - Each benchmark has its own download link to its specific trajectory archive - Removed Download from Overall aggregate table (doesn't make sense there) - Added download icon explanation to legend on benchmark tables - Column header shows just ⬇️ icon to save space Co-authored-by: openhands <openhands@all-hands.dev>4d0ae13 -
Add Download column for trajectory archives and increase table font size
openhands +1: openhandsChanges: - Add Download column to leaderboard table with download icon (⬇️) linking to trajectory archives - Preserve full_archive URLs from score data even when pydantic validation is used - Increase table font size from 14px to 15px for better readability - Add Download column documentation to tooltip descriptions Co-authored-by: openhands <openhands@all-hands.dev>b5317d7 -
Fix Pareto frontier calculation and display
openhands +1: openhandsTwo bugs were causing all models to show the frontier trophy icon: 1. The 'id' field was set to just the SDK version (e.g., 'v1.8.3') instead of a unique identifier. This caused multiple models to match when checking if they're on the Pareto frontier. Fix: Use agent_id (version_model) as the unique identifier. 2. get_pareto_df() was using the first Cost/Score columns it found (e.g., 'Swe-Bench-Multimodal Cost') instead of 'Average Cost' and 'Average Score' which should be used for the Overall leaderboard. Fix: Default to Average Cost/Score columns when available. Now only the true Pareto optimal models receive the trophy icon: - deepseek-v3.2-reasoner (lowest cost) - gemini-3-flash (best score at ~$0.54) - gpt-5.2 (best score at ~$0.86) - claude-4.5-opus (highest score) Co-authored-by: openhands <openhands@all-hands.dev>1739efc
-
Remove multi-swe-bench from OpenHands Index
openhands- Remove multi-swe-bench from benchmark mappings in simple_data_loader.py - Remove multi-swe-bench from generate_mock_jsonl.py - Update FRONTEND_DEVELOPMENT_DESCRIPTION to reference SWE-bench Multimodal (not Multi-SWE-bench) - Remove multi-swe-bench from about.py benchmark lists and acknowledgements - Update DATA_STRUCTURE.md to remove multi-swe-bench references - Keep 5 categories (Frontend uses swe-bench-multimodal, not multi-swe-bench)b978a6b
-
Update DeepSeek logo, tooltip format, and category names
openhands +1: openhands1. Replace DeepSeek logo with official SVG from LobeHub 2. Update tooltip format to: {lm_name} (SDK {version}) - Average Score, Average Cost, Openness 3. Rename categories: - Bug Fixing -> Issue Resolution - App Creation -> Greenfield - Frontend Development -> Frontend - Test Generation -> Testing - Information Gathering (unchanged) 4. Update intro paragraph to mention all category names Co-authored-by: openhands <openhands@all-hands.dev>5778893
-
Replace total_cost with cost_per_instance (average cost per instance)
openhands +1: openhands- Updated simple_data_loader.py to read cost_per_instance instead of total_cost - Changed aggregation to calculate average cost instead of sum - Updated leaderboard_transformer.py column mappings - Updated ui_components.py tooltip descriptions - Updated generate_mock_jsonl.py for mock data generation Co-authored-by: openhands <openhands@all-hands.dev>b1f3e49 -
Fix TypeError when summing costs with None values
openhands +1: openhandsFilter out None values before summing costs to prevent TypeError. Co-authored-by: openhands <openhands@all-hands.dev>cdd40ba
-
feat: Update leaderboard calculations and add incomplete entries toggle
openhands +1: openhandsChanges: 1. Rename 'OpenHands Version' to 'SDK Version' 2. Rename 'Overall Score' to 'Average Score' - now divides by 5 regardless of categories completed (missing categories count as 0) 3. Rename 'Overall Cost' to 'Total Cost' - now sums all category costs 4. Add 'Show incomplete entries' toggle (default: hidden) to filter entries that don't have all 5 categories submitted 5. Fix agent grouping to use version+model combination instead of just version (fixes issue where different models with same SDK version were incorrectly merged) 6. Fix model_dump() to use mode='json' for proper enum serialization 7. Track 'categories_completed' count per agent Co-authored-by: openhands <openhands@all-hands.dev>5998027 -
feat: Use pydantic schema models from openhands-index-results for validation
openhands +1: openhands- Keep full repo clone instead of just results/ directory - Import Metadata and ScoreEntry models from validate_schema.py - Validate agent data during loading with graceful fallback - Log validation errors to surface data quality issues This ensures data is validated against the same schema used by the results repo, catching invalid entries early. Co-authored-by: openhands <openhands@all-hands.dev>f0339f3
-
Update fallback category mappings: place SWE-Bench Multimodal under 'Frontend Development' and Swt-Bench under 'Test Generation'.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsb42a4fe -
Move commit0 to 'App Creation' category in fallback mappings.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsb16f7da -
CRITICAL FIX: Add fallback category mappings for data without agenteval.json
openhands +1: openhandsROOT CAUSE: The app was calculating 0.0 scores because: 1. No agenteval.json file exists in the data directory 2. Without it, benchmark_to_categories dict was empty 3. Category aggregation was skipped (line 162-167 in simple_data_loader.py) 4. Overall score calculation received empty list, resulted in None/0.0 SOLUTION: Add hard-coded fallback mappings when agenteval.json is missing: - swe-bench, swe-bench-multimodal, commit0, multi-swe-bench → Bug Fixing - swt-bench → Frontend Development - gaia → Information Gathering This ensures category aggregation works even without config file. Tested locally: ✅ Shows real scores (57.767, 54.692, etc.) Co-authored-by: openhands <openhands@all-hands.dev>b4ac443 -
Add debug logging to track data loading on HuggingFace Space
openhands +1: openhands- Add data integrity check in setup_data.py - Add sample data logging in simple_data_loader.py - This will help identify where the zero scores are coming from Co-authored-by: openhands <openhands@all-hands.dev>044cdf4 -
Fix score calculation to match AstaBench methodology and update categories
openhandsCritical fixes: - Changed overall score calculation from simple average of all benchmarks to macro-average of category averages - This ensures each category contributes equally regardless of benchmark count - Updated Categories Attempted from 4 to 5 categories (Bug Fixing, Frontend Dev, App Creation, Test Gen, Info Gathering) - Updated category list in leaderboard_transformer.py line 281 - Updated tooltip text to reflect 5 categories This matches the AstaBench calculation methodology.e734bf6 -
Fix agent_version display and make Overall Score bold
openhands- Fixed case mismatch in simple_data_loader.py: 'openhands version' -> 'Openhands version' - Fixed case for 'Language model' -> 'Language model' to match transformer expectations - Modified format_score_column to make Overall Score values bold - Overall Cost remains non-bold as requested0e14c25 -
Rename columns: Agent→OpenHands Version, Models Used→Language Model, remove Submitter
openhands +1: openhands- Updated simple_data_loader.py to use new column names - Updated leaderboard_transformer.py with fixed_mappings and scatter plot references - Updated ui_components.py tooltips and column handling - Changed metadata.json: agent_name→agent_version across all mock data - Added 'Bug Fixing' tag to swe-bench benchmarks in scores.json - Removed 'Submitter' column from base_cols and column references Co-authored-by: openhands <openhands@all-hands.dev>376500e -
Cleanup codebase: remove unused code, simplify data loading, and add pre-release notice
openhands +1: openhands- Remove unused github_data_loader.py (71 lines of dead code) - Clean up debug logging in app.py (replaced ~40 print statements with proper logger) - Remove scheduler code and APScheduler dependency from requirements.txt - Simplify simple_data_loader.py by removing old JSONL format fallback - Update setup_data.py to fetch data from results/ directory only - Update mock data to use new agent-centric directory structure - Add pre-release notice to main page intro paragraph Co-authored-by: openhands <openhands@all-hands.dev>855423e -
Simplify leaderboard to open vs closed models only
openhandsRemove agent tooling distinction from the leaderboard: - Simplified openness from 4 categories to 2: 'open' and 'closed' - Removed all 'tool_usage' references from data structures - Updated UI to remove Agent Tooling column and icons - Simplified scatter plots to use only 2 colors (open/closed) - Updated legends to show Model Openness instead of Agent Openness/Tooling - Regenerated mock data without tool_usage field - Added openness normalization to handle legacy values Files modified: - aliases.py: Simplified openness constants, removed tool usage - simple_data_loader.py: Added openness normalization, removed tool_usage loading - leaderboard_transformer.py: Removed Agent Tooling column, simplified color mapping - ui_components.py: Removed tooling icons/tooltips, simplified to openness only - generate_mock_jsonl.py: Removed tool_usage from mock data generationca754bb
-
Update data loader to support agent-centric directory structure
openhands +1: openhands- Add support for new results/YYYYMMDD_model/ directory structure - Each agent directory contains metadata.json and scores.json - Maintain backward compatibility with old JSONL format - Update mock data to use new structure Co-authored-by: openhands <openhands@all-hands.dev>e003f7b -
Update UI: All-Hands-AI color scheme, agent version column names, and OpenHands logo
openhands +1: openhands- Apply All-Hands-AI color scheme (yellow accent, indigo links, beige background) - Change 'Agent' column to 'Agent Version' - Remove 'Submitter' column - Change 'Models Used' to 'Model' - Update mock data with version numbers (1.0.2, 1.0.1, etc.) - Replace logo with OpenHands branding Co-authored-by: openhands <openhands@all-hands.dev>0ee2099 -
Fix category cost calculation - add category-level aggregation
openhands +1: openhands- Updated simple_data_loader.py to build proper category-to-benchmarks mapping from agenteval.json - Added logic to calculate category-level aggregate scores and costs from individual benchmark results - Updated agenteval.json to include category tags (Bug Fixing, App Creation, etc.) - Fixes issue where Bug Fixing cost was showing as 0.0 instead of the average of swe-bench and swe-bench-multimodal costs Co-authored-by: openhands <openhands@all-hands.dev>5e9c3b9 -
Fix null columns by mapping source data to expected column names
openhands- Modified simple_data_loader.py to use lowercase column names that match transformer expectations - Changed agent_name → agent, llm_base → models used, tool_usage → agent tooling, submission_time → date - Fixed leaderboard_transformer.py tag map capitalization to match column names - All 24 columns now populated correctly (no null values) - Resolves JavaScript table rendering erroraa07520 -
Convert to JSONL data format and remove agent-eval dependency
openhands +1: openhands- Replace parquet files with JSONL for better readability and simplicity - Remove agent-eval library dependency to avoid secret requirements - Create simple_data_loader.py as lightweight replacement for LeaderboardViewer - Create submission_utils.py for submission handling functions - Simplify Dockerfile (no SSH keys or secrets needed) - Reduce requirements.txt to essential packages only - Generate mock data for all 6 benchmarks (swe-bench, multi-swe-bench, swe-bench-multimodal, swt-bench, commit0, gaia) - Update aliases.py to define constants locally - Fix imports in submission.py and test files Co-authored-by: openhands <openhands@all-hands.dev>1027cfb