Spaces:
Running
Running
Commit History
-
Move alternative agents to a dedicated page
Verified signatureAddress review feedback from the original commit: - Mixing default OpenHands runs and alternative-agent runs (Claude Code, Codex, Gemini CLI, OpenHands Sub-agents) in the same leaderboard table and the same scatter plot is misleading. Their cost/runtime numbers aren't apples-to-apples across harnesses, and conflating them with the OpenHands ranking on Home obscures what the index is actually measuring. - Move alternative-agent rows to a dedicated /alternative-agents page. Home and the per-category subpages stay exactly as before (default-OpenHands only). Changes: - simple_data_loader.SimpleLeaderboardViewer now takes an `agent_filter` argument, defaulting to "openhands". When "openhands" it walks results/{model}/. When "alternative" it walks alternative_agents/{type}/{model}/. - ui_components.get_leaderboard_viewer_instance / get_full_leaderboard_data thread `agent_filter` through and key the in-memory cache on (split, agent_filter) so the two pages don't fig…d7034b8
-
Hide incomplete entries from home page charts by default
openhands0b126a8
-
fix: Add Visualization column to main table (not just benchmark tables)
openhands +1: openhandsThe previous commit only added visualization links to individual benchmark detail tables. This commit adds the Visualization column to the main front page table as well. Changes: - Add 'visualization' column to record in simple_data_loader.py - Add 'Visualization' to ending_cols in leaderboard_transformer.py - Add make_visualization_icons() to format the column in ui_components.py - Update column widths and datatypes to include Visualization - Add tooltip description for Visualization column Co-authored-by: openhands <openhands@all-hands.dev>5f628a6 -
feat: Add Visualization column for Laminar eval links
openhands +1: openhandsDisplay eval_visualization_page URLs as clickable 📊 icons in a new Visualization column next to Logs. Changes: - Load eval_visualization_page from score entries in simple_data_loader.py - Store as '{benchmark} visualization' column similar to download URLs - Add 📊 column to benchmark detail tables with clickable links - Only show icon if visualization URL exists, otherwise blank - Add tooltip documentation for the new column Co-authored-by: openhands <openhands@all-hands.dev>dfa8bfc
-
Widen Logs column to prevent vertical stacking of download icons
openhands +1: openhandsCo-authored-by: openhands <openhands@all-hands.dev>9cab912 -
Add download icons to Logs column for benchmark results
openhands +1: openhandsCo-authored-by: openhands <openhands@all-hands.dev>67867ec
-
Format date column to show only date, not time
openhands- Add format_date_column function to leaderboard_transformer.py - Apply date formatting in ui_components.py for all table displays - Strips time portion from ISO datetime strings (e.g. 2025-11-24T19:56:00 -> 2025-11-24)2b7cd27
-
Fix: Preserve mark_by selection during periodic data refresh
openhands +1: openhandsThe 60-second timer that checks for data updates was not passing the current mark_by dropdown value, causing charts to revert to 'Company' when data refreshed. Changes: - Add mark_by parameter to check_and_refresh_data() - Add mark_by parameter to check_and_refresh_all() - Add mark_by parameter to check_and_refresh_simple() - Include mark_by_dropdown in timer inputs for all refresh paths - Pass mark_by to create_cost_scatter_plot() and create_runtime_scatter_plot() Co-authored-by: openhands <openhands@all-hands.dev>df058f7 -
Add 'Mark systems by' selector for scatter plot icons (Company/Openness/Country)
openhands +1: openhands- Add MARK_BY_CHOICES and MARK_BY_DEFAULT constants - Add country flag SVGs (US, China, France) - Add COUNTRY_FLAG_MAP and MODEL_COUNTRY_MAP for country detection - Add get_openness_icon() and get_country_from_model() functions - Add get_marker_icon() to select appropriate icon based on mark_by setting - Modify _plot_scatter_plotly() to accept mark_by parameter - Add mark_by_dropdown to UI in ui_components.py - Update update_display() to regenerate plots dynamically based on mark_by selection Co-authored-by: openhands <openhands@all-hands.dev>ed6e90d -
Clean up unused code, files, and assets
openhandsRemoved unused Python code: - svg_to_data_uri() from leaderboard_transformer.py - create_svg_html() from ui_components.py - format_llm_base_with_html() from ui_components.py Consolidated duplicate code: - Renamed get_company_from_model_name to get_company_from_model in leaderboard_transformer.py - Removed duplicate COMPANY_LOGO_MAP and get_company_from_model from ui_components.py - ui_components.py and visualizations.py now import from leaderboard_transformer.py Removed unused files: - submission.py (submission functionality not exposed in app) - submission_utils.py (only used by submission.py) - generate_mock_jsonl.py (development utility) - tests/integration/test_submission.py (tests for removed submission code) Removed unused config: - CONTACT_DATASET and SUBMISSION_DATASET from config.py Removed unused CSS: - Submission page CSS section from content.py Removed 35 unused asset files: - Various legend SVGs (api-*, c-*, os-*, custom-*, standard-*, equivalent-*) - Category icons (…bb0cd90 -
Only show 'Detailed Benchmark Results' when more than one benchmark exists
openhands- Modified create_benchmark_details_display to check benchmark count - If category has 1 or fewer benchmarks, skip showing the detailed section - Updated docstring to reflect this behaviord17eff0 -
Connect 'Show only open models' checkbox to Winners and Evolution sections
openhands- Modified create_leaderboard_display to return show_open_only_checkbox - Updated main_page.py to connect checkbox to Winners by Category and Evolution charts - Pre-generate both all and open-only versions of Winners HTML and Evolution plot - When checkbox is toggled, both sections now update to show only open models49f9739
-
Winners by Category: put scores before names
openhands +1: openhands- Swap column order so score appears first, then model name - Update CSS styling for new column order Co-authored-by: openhands <openhands@all-hands.dev>24ff7a3 -
Use emojis instead of images in Winners by Category table
openhands +1: openhands- Replace category SVG icons with emojis (🐛 🌱 🎨 🧪 🔍) - Remove company logo images from model cells - Move medal emojis to after model name (e.g., 'claude-4.5-opus 🥇') Co-authored-by: openhands <openhands@all-hands.dev>63c73f3 -
Refactor Winners by Category to single unified table
openhands +1: openhands- Changed from multiple category cards to a single horizontal table - Format: | [emoji] Category | Score | ... for each category - Display only model name (removed SDK version) - Deduplicate by model name, keeping highest score - Updated CSS for new table layout with responsive design Co-authored-by: openhands <openhands@all-hands.dev>4f4eb00 -
Add Winners by Category section to main page
openhands +1: openhands- Implemented new 'Winners by Category' section displaying top 5 systems for each benchmark category - Added functions get_winners_by_category() and create_winners_by_category_html() to ui_components.py - Created responsive grid layout with category cards showing: - Category icon and name header - Ranked table with medal emojis (🥇🥈🥉) for top 3 - Company logos for model providers - SDK version and model name - Category score - Added comprehensive CSS styles in content.py: - Light and dark mode support - Responsive grid for mobile devices - Hover effects for cards - Consistent styling with existing design system - Section placed directly after main leaderboard results Co-authored-by: openhands <openhands@all-hands.dev>c14a283 -
Add 'Show only open models' checkbox filter
openhands +1: openhands- Added checkbox to filter leaderboard table and scatter plots to show only open-source/open-weight models - Checkbox shows count of open and closed models (e.g., '4 open, 5 closed') - Filter works independently and in combination with 'Show incomplete entries' checkbox - Filter applies to both Cost/Performance and Runtime/Performance scatter plots - Updated timer/refresh logic to maintain filter state on data refresh Co-authored-by: openhands <openhands@all-hands.dev>5cdf97c -
Add runtime column and Cost/Performance + Runtime/Performance charts to all pages
openhands +1: openhands- Add runtime tracking to data loader (simple_data_loader.py) - Add format_runtime_column function (leaderboard_transformer.py) - Update _plot_scatter_plotly with plot_type parameter ('cost' or 'runtime') - Update chart titles to 'OpenHands Index XXX Cost/Performance' and 'OpenHands Index XXX Runtime/Performance' - Add Runtime column to main leaderboard and benchmark details tables - Display both cost/performance and runtime/performance scatter plots on all pages Co-authored-by: openhands <openhands@all-hands.dev>2854ddd
-
Move Download column to benchmark-specific tables only
openhands +1: openhands- Download column now appears on benchmark-specific leaderboard tables (e.g., SWE-Bench, GAIA) - Each benchmark has its own download link to its specific trajectory archive - Removed Download from Overall aggregate table (doesn't make sense there) - Added download icon explanation to legend on benchmark tables - Column header shows just ⬇️ icon to save space Co-authored-by: openhands <openhands@all-hands.dev>4d0ae13 -
Add Download column for trajectory archives and increase table font size
openhands +1: openhandsChanges: - Add Download column to leaderboard table with download icon (⬇️) linking to trajectory archives - Preserve full_archive URLs from score data even when pydantic validation is used - Increase table font size from 14px to 15px for better readability - Add Download column documentation to tooltip descriptions Co-authored-by: openhands <openhands@all-hands.dev>b5317d7
-
Add timer-based auto-refresh for leaderboard data
openhandsWhen background data refresh occurs, the UI now automatically updates within 60 seconds without requiring a page reload. This fixes the issue where new results pushed to the GitHub repo weren't reflected on the live site until the HF Space container was restarted. Changes: - Add data version tracking to detect when cache is cleared - Add gr.Timer() that checks for data updates every 60 seconds - Automatically refresh DataFrame and Plot components when new data is available974f31f
-
Move 'Show incomplete entries' checkbox above plot and apply filter to both
openhands- Moved the checkbox above the scatter plot for better UX - The checkbox now filters both the scatter plot AND the table - Pre-computed both complete and all entries scatter plots for fast toggling - When checkbox is toggled, both plot and table update simultaneously361b5c2 -
Add periodic cache refresh for leaderboard data
openhands- Add background refresh scheduler that checks for new data every hour - Implement thread-safe caching with configurable TTL (CACHE_TTL_SECONDS env var) - Register cache clear callback to invalidate viewer cache when data refreshes - Add refresh_data_if_needed() function for on-demand refresh - Data is fetched from GitHub repo and cached, then periodically refreshed This ensures the leaderboard shows fresh data while maintaining good performance through caching.6bddf26
-
Update DeepSeek logo, tooltip format, and category names
openhands +1: openhands1. Replace DeepSeek logo with official SVG from LobeHub 2. Update tooltip format to: {lm_name} (SDK {version}) - Average Score, Average Cost, Openness 3. Rename categories: - Bug Fixing -> Issue Resolution - App Creation -> Greenfield - Frontend Development -> Frontend - Test Generation -> Testing - Information Gathering (unchanged) 4. Update intro paragraph to mention all category names Co-authored-by: openhands <openhands@all-hands.dev>5778893 -
Fix table icons layout and add Qwen/MiniMax logos
openhands +1: openhands1. Fixed table icons to display horizontally using flex container 2. Added proper Qwen logo (purple star) replacing Alibaba logo 3. Added MiniMax logo (pink waveform) 4. Updated COMPANY_LOGO_MAP in both files Co-authored-by: openhands <openhands@all-hands.dev>72b86cb -
UI cleanup and About page updates
openhands +1: openhands1. Removed legend box from graph (OpenHands/Pareto/Company Logos) 2. Combined Pareto, lock, and company logo columns into Language Model column 3. Removed dataset structure SVG image from main page 4. Updated Acknowledgements section - shortened AstaBench credit, added links to component benchmarks 5. Updated Citation to be for OpenHands Index with authors: Juan Michelini, Simon Rosenberg, Xingyao Wang, Graham Neubig Co-authored-by: openhands <openhands@all-hands.dev>6737ff3 -
Multiple graph and table improvements
openhands +1: openhands- Removed 'Missing Cost Data' vertical divider line from graph - Added log scale to x-axis (cost) for better data visualization - Created custom SVG lock icons: blue open lock, red closed lock - Replaced lock emojis with custom SVG lock icons in tables - Made graph icons smaller for cleaner appearance - Moved model name labels higher above icons using annotations Co-authored-by: openhands <openhands@all-hands.dev>fcb3d0b -
Replace open/closed model distinction with lock emojis in tables
openhands +1: openhands- Changed from colored ellipse SVG icons to 🔓 (open lock) and 🔒 (closed lock) emojis - Updated main leaderboard table and benchmark tables - Updated legend to show lock emojis instead of colored dots - Updated tooltip content to use lock emoji icons Co-authored-by: openhands <openhands@all-hands.dev>8a3a9eb -
Remove open/closed distinction from graph, use company logos as data points
openhands +1: openhands- Replaced colored markers (pink/yellow) with company logo images on scatter plots - Downloaded official logos from SimpleIcons CDN (Anthropic, Google, Meta, HuggingFace) - Created recognizable logos for OpenAI, Mistral, DeepSeek, Cohere, Moonshot - Updated graph legend to show 'Company Logos' instead of 'Model Openness' - Maintained invisible hover markers for tooltip functionality - Model names still labeled on Pareto frontier points - Tables still show company logos in first column Co-authored-by: openhands <openhands@all-hands.dev>b6ec318 -
Add company logos to graphs and tables, label frontier points with model names
openhands +1: openhands- Created SVG logo files for major AI companies (Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, xAI, Cohere, Alibaba, Moonshot) - Added company logo mapping based on model name patterns - Added Company logo column to both main leaderboard and benchmark tables - Added company logo images on scatter plots at each data point - Added model name labels to Pareto frontier points on graphs - Updated column widths to accommodate new Company column Co-authored-by: openhands <openhands@all-hands.dev>800e404
-
Replace total_cost with cost_per_instance (average cost per instance)
openhands +1: openhands- Updated simple_data_loader.py to read cost_per_instance instead of total_cost - Changed aggregation to calculate average cost instead of sum - Updated leaderboard_transformer.py column mappings - Updated ui_components.py tooltip descriptions - Updated generate_mock_jsonl.py for mock data generation Co-authored-by: openhands <openhands@all-hands.dev>b1f3e49
-
fix: Column naming and incomplete entries toggle
openhands +1: openhandsFixes: 1. Change 'Overall Score' to 'Average Score' in all places (including view() method which was constructing 'Overall Score' directly) 2. Change 'Overall Cost' to 'Total Cost' - properly route to Total Cost column for Overall view 3. Fix incomplete entries toggle: - Check 'Categories Attempted' column (formatted as 'X/5') - Filter on '5/5' for complete entries - If no complete entries exist, show all entries with explanatory note - Keep 'Categories Attempted' column visible in the table Co-authored-by: openhands <openhands@all-hands.dev>4ab5f97 -
feat: Update leaderboard calculations and add incomplete entries toggle
openhands +1: openhandsChanges: 1. Rename 'OpenHands Version' to 'SDK Version' 2. Rename 'Overall Score' to 'Average Score' - now divides by 5 regardless of categories completed (missing categories count as 0) 3. Rename 'Overall Cost' to 'Total Cost' - now sums all category costs 4. Add 'Show incomplete entries' toggle (default: hidden) to filter entries that don't have all 5 categories submitted 5. Fix agent grouping to use version+model combination instead of just version (fixes issue where different models with same SDK version were incorrectly merged) 6. Fix model_dump() to use mode='json' for proper enum serialization 7. Track 'categories_completed' count per agent Co-authored-by: openhands <openhands@all-hands.dev>5998027
-
Fix UI score formatting: do not coerce NaN to 0; rely on format_score_column to show 'Not Submitted'.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsc68aa7d -
Fix data plotting requirements and server port handling; ensure per-benchmark plots use correct agent column.\n\n- Respect HOST/PORT env for local runs\n- Use 'OpenHands Version' in plot requirements\n- Avoid plotting when use_plotly=False\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsfb3d0db -
Remove unused AstaBench category files and update UI to OpenHands categories
openhandsCleanup actions: - Deleted 4 unused AstaBench category files: c_and_e.py, data_analysis.py, e2e.py, literature_understanding.py - Updated ui_components.py tooltips to show OpenHands categories (Bug Fixing, Frontend Development, etc.) instead of AstaBench categories - Changed tooltip from 'four categories' to 'five categories' - Updated leaderboard_transformer.py comment to reflect actual OpenHands field mappings - Category list now: Bug Fixing, Frontend Development, App Creation, Test Generation, Information Gathering This removes ~1KB of dead code and fixes user-facing tooltips to match actual categories.6a0d1cb -
Fix score calculation to match AstaBench methodology and update categories
openhandsCritical fixes: - Changed overall score calculation from simple average of all benchmarks to macro-average of category averages - This ensures each category contributes equally regardless of benchmark count - Updated Categories Attempted from 4 to 5 categories (Bug Fixing, Frontend Dev, App Creation, Test Gen, Info Gathering) - Updated category list in leaderboard_transformer.py line 281 - Updated tooltip text to reflect 5 categories This matches the AstaBench calculation methodology.e734bf6 -
Swap column order and fix duplicate column warnings
openhands +1: openhands- Swapped Language Model and OpenHands Version column order - Fixed duplicate column warning by renaming columns before getting headers - Updated mock data with proper agent_version values (1.0.1, 1.0.2) - Removed duplicate agent_version keys from metadata.json files Co-authored-by: openhands <openhands@all-hands.dev>3781804 -
Rename columns: Agent→OpenHands Version, Models Used→Language Model, remove Submitter
openhands +1: openhands- Updated simple_data_loader.py to use new column names - Updated leaderboard_transformer.py with fixed_mappings and scatter plot references - Updated ui_components.py tooltips and column handling - Changed metadata.json: agent_name→agent_version across all mock data - Added 'Bug Fixing' tag to swe-bench benchmarks in scores.json - Removed 'Submitter' column from base_cols and column references Co-authored-by: openhands <openhands@all-hands.dev>376500e -
Force rebuild: Update comment
openhands64c8899 -
Fix missing get_combined_icon_html reference in main page
openhands- Fixed remaining reference to removed get_combined_icon_html function - Now using get_openness_icon_html inline function instead - Tested locally and app runs successfully without errorsae0bf64 -
Simplify leaderboard to open vs closed models only
openhandsRemove agent tooling distinction from the leaderboard: - Simplified openness from 4 categories to 2: 'open' and 'closed' - Removed all 'tool_usage' references from data structures - Updated UI to remove Agent Tooling column and icons - Simplified scatter plots to use only 2 colors (open/closed) - Updated legends to show Model Openness instead of Agent Openness/Tooling - Regenerated mock data without tool_usage field - Added openness normalization to handle legacy values Files modified: - aliases.py: Simplified openness constants, removed tool usage - simple_data_loader.py: Added openness normalization, removed tool_usage loading - leaderboard_transformer.py: Removed Agent Tooling column, simplified color mapping - ui_components.py: Removed tooling icons/tooltips, simplified to openness only - generate_mock_jsonl.py: Removed tool_usage from mock data generationca754bb
-
Fix runtime errors and data loading
openhands- Created combined test.jsonl with all 6 benchmark results - Updated agenteval.json to include test split with all tasks - Added missing imports to submission.py (DatasetDict, Dataset, etc.) - Fixed LeaderboardViewer reference in ui_components.py - Commented out submission page due to Modal component issue - Disabled scheduler that was causing indentation issues66fcacd -
Convert to JSONL data format and remove agent-eval dependency
openhands +1: openhands- Replace parquet files with JSONL for better readability and simplicity - Remove agent-eval library dependency to avoid secret requirements - Create simple_data_loader.py as lightweight replacement for LeaderboardViewer - Create submission_utils.py for submission handling functions - Simplify Dockerfile (no SSH keys or secrets needed) - Reduce requirements.txt to essential packages only - Generate mock data for all 6 benchmarks (swe-bench, multi-swe-bench, swe-bench-multimodal, swt-bench, commit0, gaia) - Update aliases.py to define constants locally - Fix imports in submission.py and test files Co-authored-by: openhands <openhands@all-hands.dev>1027cfb -
Initial OpenHands Index leaderboard based on ASTA Bench
openhands +1: openhands- Updated branding (title, README, logo) - Changed dataset references to OpenHands Index datasets - Modified agenteval.json to include 6 OpenHands datasets: * swe-bench * multi-swe-bench * swe-bench-multimodal * swt-bench * commit0 * gaia - Added mock data for initial testing Co-authored-by: openhands <openhands@all-hands.dev>085a012