Spaces:
Running
Running
Commit History
-
Empty PR to test CI
openhands +1: openhandsCo-authored-by: openhands <openhands@all-hands.dev>da83a25 -
Update Frontend description to use SWE-bench Multimodal (Verified)
openhands +1: openhands- Clarify that Frontend category uses the dev set of SWE-bench Multimodal - Note that it uses only problems verified as solveable by human review - Add links to the dataset and solveability annotations Co-authored-by: openhands <openhands@all-hands.dev>eb1e409
-
Move Download column to benchmark-specific tables only
openhands +1: openhands- Download column now appears on benchmark-specific leaderboard tables (e.g., SWE-Bench, GAIA) - Each benchmark has its own download link to its specific trajectory archive - Removed Download from Overall aggregate table (doesn't make sense there) - Added download icon explanation to legend on benchmark tables - Column header shows just ⬇️ icon to save space Co-authored-by: openhands <openhands@all-hands.dev>4d0ae13 -
Add Download column for trajectory archives and increase table font size
openhands +1: openhandsChanges: - Add Download column to leaderboard table with download icon (⬇️) linking to trajectory archives - Preserve full_archive URLs from score data even when pydantic validation is used - Increase table font size from 14px to 15px for better readability - Add Download column documentation to tooltip descriptions Co-authored-by: openhands <openhands@all-hands.dev>b5317d7 -
Fix Pareto frontier calculation and display
openhands +1: openhandsTwo bugs were causing all models to show the frontier trophy icon: 1. The 'id' field was set to just the SDK version (e.g., 'v1.8.3') instead of a unique identifier. This caused multiple models to match when checking if they're on the Pareto frontier. Fix: Use agent_id (version_model) as the unique identifier. 2. get_pareto_df() was using the first Cost/Score columns it found (e.g., 'Swe-Bench-Multimodal Cost') instead of 'Average Cost' and 'Average Score' which should be used for the Overall leaderboard. Fix: Default to Average Cost/Score columns when available. Now only the true Pareto optimal models receive the trophy icon: - deepseek-v3.2-reasoner (lowest cost) - gemini-3-flash (best score at ~$0.54) - gpt-5.2 (best score at ~$0.86) - claude-4.5-opus (highest score) Co-authored-by: openhands <openhands@all-hands.dev>1739efc -
Fix: Treat zero scores as valid results, not missing categories
openhands +1: openhandsA score of 0.0 is a legitimate evaluation result (e.g., qwen-3-coder's commit0 score). Previously, entries with any 0.0 score were incorrectly marked as 'incomplete' because the code treated 0.0 the same as missing. This fix ensures that only truly missing values (NA/None) are considered incomplete, while 0.0 scores are counted as attempted categories. Co-authored-by: openhands <openhands@all-hands.dev>eff7f34 -
Reduce cache TTL from 1 hour to 15 minutes for faster data refresh
openhands +1: openhandsCo-authored-by: openhands <openhands@all-hands.dev>da96293
-
Remove multi-swe-bench from OpenHands Index
openhands- Remove multi-swe-bench from benchmark mappings in simple_data_loader.py - Remove multi-swe-bench from generate_mock_jsonl.py - Update FRONTEND_DEVELOPMENT_DESCRIPTION to reference SWE-bench Multimodal (not Multi-SWE-bench) - Remove multi-swe-bench from about.py benchmark lists and acknowledgements - Update DATA_STRUCTURE.md to remove multi-swe-bench references - Keep 5 categories (Frontend uses swe-bench-multimodal, not multi-swe-bench)b978a6b -
Add timer-based auto-refresh for leaderboard data
openhandsWhen background data refresh occurs, the UI now automatically updates within 60 seconds without requiring a page reload. This fixes the issue where new results pushed to the GitHub repo weren't reflected on the live site until the HF Space container was restarted. Changes: - Add data version tracking to detect when cache is cleared - Add gr.Timer() that checks for data updates every 60 seconds - Automatically refresh DataFrame and Plot components when new data is available974f31f
-
Move 'Show incomplete entries' checkbox above plot and apply filter to both
openhands- Moved the checkbox above the scatter plot for better UX - The checkbox now filters both the scatter plot AND the table - Pre-computed both complete and all entries scatter plots for fast toggling - When checkbox is toggled, both plot and table update simultaneously361b5c2 -
Add periodic cache refresh for leaderboard data
openhands- Add background refresh scheduler that checks for new data every hour - Implement thread-safe caching with configurable TTL (CACHE_TTL_SECONDS env var) - Register cache clear callback to invalidate viewer cache when data refreshes - Add refresh_data_if_needed() function for on-demand refresh - Data is fetched from GitHub repo and cached, then periodically refreshed This ensures the leaderboard shows fresh data while maintaining good performance through caching.6bddf26
-
Change blue links and text to dark grey (#555555)
openhands +1: openhandsUpdated link colors in both CSS variables and Gradio theme to use dark grey instead of blue/indigo for better visual consistency. Co-authored-by: openhands <openhands@all-hands.dev>d68f190 -
Remove illustrative figures from category sub-pages
openhands +1: openhandsCo-authored-by: openhands <openhands@all-hands.dev>d2c3c09 -
Update DeepSeek logo, tooltip format, and category names
openhands +1: openhands1. Replace DeepSeek logo with official SVG from LobeHub 2. Update tooltip format to: {lm_name} (SDK {version}) - Average Score, Average Cost, Openness 3. Rename categories: - Bug Fixing -> Issue Resolution - App Creation -> Greenfield - Frontend Development -> Frontend - Test Generation -> Testing - Information Gathering (unchanged) 4. Update intro paragraph to mention all category names Co-authored-by: openhands <openhands@all-hands.dev>5778893 -
Fix frontier labels to use log10 coordinates for log scale axis
openhands +1: openhandsAnnotations in Plotly need log10(x) coordinates when the x-axis uses log scale (see Plotly issue #2580). Updated the frontier label annotations to transform x values to log10. Co-authored-by: openhands <openhands@all-hands.dev>8cdce51 -
Method C: Use domain coordinates for layout images with log scale
openhands +1: openhandsLayout images don't work correctly with log scale axes when using data coordinates (xref='x'). The solution is to use domain coordinates (xref='x domain') which represent positions as fractions (0-1) of the plot area. This requires: 1. Manually calculating the log10 position of each point 2. Converting to domain coordinates relative to the axis range 3. Setting explicit axis ranges that match the domain calculations Tested locally - colored circles now align perfectly with scatter markers. Co-authored-by: openhands <openhands@all-hands.dev>4a4a7b5 -
Method B: Use raw data values for layout images (revert log transformation)
openhands +1: openhandsTesting whether Plotly handles log transformation automatically for layout images the same way it does for scatter traces. Co-authored-by: openhands <openhands@all-hands.dev>46ff970 -
Method A: Fix logo positions for log scale x-axis
openhands +1: openhandsTransform image x-coordinates to log10(x) since Plotly layout images expect log-scale coordinates when the axis type is 'log'. Also adjusted logo size to work better with log scale units. Co-authored-by: openhands <openhands@all-hands.dev>7b4a3a1 -
Enable log scale for x-axis (cost) in graphs
openhands +1: openhandsAdded type='log' to xaxis configuration in Plotly charts. This improves readability when cost values span a wide range. Co-authored-by: openhands <openhands@all-hands.dev>5d32e7b -
Trigger rebuild
openhands8d4cfda -
Update intro text to focus on motivation rather than metrics
openhands +1: openhandsSimplified the above-the-fold description to emphasize: - Why the index exists (need for standardized agent evaluation) - What it provides (aggregated performance + cost view) - Who benefits (developers and researchers) Moved detailed metric explanations to the About page. Co-authored-by: openhands <openhands@all-hands.dev>369c590 -
Fix table icons layout and add Qwen/MiniMax logos
openhands +1: openhands1. Fixed table icons to display horizontally using flex container 2. Added proper Qwen logo (purple star) replacing Alibaba logo 3. Added MiniMax logo (pink waveform) 4. Updated COMPANY_LOGO_MAP in both files Co-authored-by: openhands <openhands@all-hands.dev>72b86cb -
UI cleanup and About page updates
openhands +1: openhands1. Removed legend box from graph (OpenHands/Pareto/Company Logos) 2. Combined Pareto, lock, and company logo columns into Language Model column 3. Removed dataset structure SVG image from main page 4. Updated Acknowledgements section - shortened AstaBench credit, added links to component benchmarks 5. Updated Citation to be for OpenHands Index with authors: Juan Michelini, Simon Rosenberg, Xingyao Wang, Graham Neubig Co-authored-by: openhands <openhands@all-hands.dev>6737ff3 -
Fix graph alignment issues
openhands +1: openhands- Reverted log scale on x-axis (caused misalignment with layout images) - Restored linear scale for cost axis - Fixed logo sizing for linear scale Co-authored-by: openhands <openhands@all-hands.dev>af81bcf -
Multiple graph and table improvements
openhands +1: openhands- Removed 'Missing Cost Data' vertical divider line from graph - Added log scale to x-axis (cost) for better data visualization - Created custom SVG lock icons: blue open lock, red closed lock - Replaced lock emojis with custom SVG lock icons in tables - Made graph icons smaller for cleaner appearance - Moved model name labels higher above icons using annotations Co-authored-by: openhands <openhands@all-hands.dev>fcb3d0b -
Replace open/closed model distinction with lock emojis in tables
openhands +1: openhands- Changed from colored ellipse SVG icons to 🔓 (open lock) and 🔒 (closed lock) emojis - Updated main leaderboard table and benchmark tables - Updated legend to show lock emojis instead of colored dots - Updated tooltip content to use lock emoji icons Co-authored-by: openhands <openhands@all-hands.dev>8a3a9eb -
Remove open/closed distinction from graph, use company logos as data points
openhands +1: openhands- Replaced colored markers (pink/yellow) with company logo images on scatter plots - Downloaded official logos from SimpleIcons CDN (Anthropic, Google, Meta, HuggingFace) - Created recognizable logos for OpenAI, Mistral, DeepSeek, Cohere, Moonshot - Updated graph legend to show 'Company Logos' instead of 'Model Openness' - Maintained invisible hover markers for tooltip functionality - Model names still labeled on Pareto frontier points - Tables still show company logos in first column Co-authored-by: openhands <openhands@all-hands.dev>b6ec318 -
Add company logos to graphs and tables, label frontier points with model names
openhands +1: openhands- Created SVG logo files for major AI companies (Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, xAI, Cohere, Alibaba, Moonshot) - Added company logo mapping based on model name patterns - Added Company logo column to both main leaderboard and benchmark tables - Added company logo images on scatter plots at each data point - Added model name labels to Pareto frontier points on graphs - Updated column widths to accommodate new Company column Co-authored-by: openhands <openhands@all-hands.dev>800e404
-
Replace total_cost with cost_per_instance (average cost per instance)
openhands +1: openhands- Updated simple_data_loader.py to read cost_per_instance instead of total_cost - Changed aggregation to calculate average cost instead of sum - Updated leaderboard_transformer.py column mappings - Updated ui_components.py tooltip descriptions - Updated generate_mock_jsonl.py for mock data generation Co-authored-by: openhands <openhands@all-hands.dev>b1f3e49 -
Fix TypeError when summing costs with None values
openhands +1: openhandsFilter out None values before summing costs to prevent TypeError. Co-authored-by: openhands <openhands@all-hands.dev>cdd40ba -
Remove Test Set/Validation Set tabs, keep single results view
openhands +1: openhandsSimplified the UI by removing the dual tab structure (Test Set / Validation Set) and keeping only a single results display using the test set data. Co-authored-by: openhands <openhands@all-hands.dev>b8aea20
-
fix: Update Total cost description in intro to be sum, not average
openhands +1: openhandsThe previous sed command didn't work due to a unicode apostrophe character. Co-authored-by: openhands <openhands@all-hands.dev>6a5c447 -
docs: Update descriptive text to use Average Score and Total Cost
openhands +1: openhands- Changed 'Overall score' to 'Average score' in content.py and about.py - Changed 'Overall cost' to 'Total cost' in content.py and about.py - Updated Total cost description to reflect that it's a sum, not an average Co-authored-by: openhands <openhands@all-hands.dev>bb0f7af -
fix: Column naming and incomplete entries toggle
openhands +1: openhandsFixes: 1. Change 'Overall Score' to 'Average Score' in all places (including view() method which was constructing 'Overall Score' directly) 2. Change 'Overall Cost' to 'Total Cost' - properly route to Total Cost column for Overall view 3. Fix incomplete entries toggle: - Check 'Categories Attempted' column (formatted as 'X/5') - Filter on '5/5' for complete entries - If no complete entries exist, show all entries with explanatory note - Keep 'Categories Attempted' column visible in the table Co-authored-by: openhands <openhands@all-hands.dev>4ab5f97 -
feat: Update leaderboard calculations and add incomplete entries toggle
openhands +1: openhandsChanges: 1. Rename 'OpenHands Version' to 'SDK Version' 2. Rename 'Overall Score' to 'Average Score' - now divides by 5 regardless of categories completed (missing categories count as 0) 3. Rename 'Overall Cost' to 'Total Cost' - now sums all category costs 4. Add 'Show incomplete entries' toggle (default: hidden) to filter entries that don't have all 5 categories submitted 5. Fix agent grouping to use version+model combination instead of just version (fixes issue where different models with same SDK version were incorrectly merged) 6. Fix model_dump() to use mode='json' for proper enum serialization 7. Track 'categories_completed' count per agent Co-authored-by: openhands <openhands@all-hands.dev>5998027 -
feat: Add open_weights to openness mapping
openhands +1: openhandsMaps the 'open_weights' value from openhands-index-results to the simplified 'open' display category. This ensures DeepSeek, Qwen, Kimi, and MiniMax models display correctly as open-source. Co-authored-by: openhands <openhands@all-hands.dev>55da48c -
feat: Use pydantic schema models from openhands-index-results for validation
openhands +1: openhands- Keep full repo clone instead of just results/ directory - Import Metadata and ScoreEntry models from validate_schema.py - Validate agent data during loading with graceful fallback - Log validation errors to surface data quality issues This ensures data is validated against the same schema used by the results repo, catching invalid entries early. Co-authored-by: openhands <openhands@all-hands.dev>f0339f3 -
Fix: Handle pd.NA values in calculate_attempted function
openhands +1: openhandsThe boolean comparison 'value not in [None, 0.0]' fails when the value is pd.NA because pandas NA doesn't support boolean operations with in/not in operators. Changed to use pd.notna() for proper NA-aware checking. Co-authored-by: openhands <openhands@all-hands.dev>28554f6
-
Update fallback category mappings: place SWE-Bench Multimodal under 'Frontend Development' and Swt-Bench under 'Test Generation'.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsb42a4fe -
Move commit0 to 'App Creation' category in fallback mappings.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsb16f7da -
Fix UI score formatting: do not coerce NaN to 0; rely on format_score_column to show 'Not Submitted'.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsc68aa7d -
Fix score formatting to avoid coercing NaN to 0; show 'Not Submitted' instead.\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhands5d82fab -
Fix data plotting requirements and server port handling; ensure per-benchmark plots use correct agent column.\n\n- Respect HOST/PORT env for local runs\n- Use 'OpenHands Version' in plot requirements\n- Avoid plotting when use_plotly=False\n\nCo-authored-by: openhands <openhands@all-hands.dev>
openhandsfb3d0db -
CRITICAL FIX: Add fallback category mappings for data without agenteval.json
openhands +1: openhandsROOT CAUSE: The app was calculating 0.0 scores because: 1. No agenteval.json file exists in the data directory 2. Without it, benchmark_to_categories dict was empty 3. Category aggregation was skipped (line 162-167 in simple_data_loader.py) 4. Overall score calculation received empty list, resulted in None/0.0 SOLUTION: Add hard-coded fallback mappings when agenteval.json is missing: - swe-bench, swe-bench-multimodal, commit0, multi-swe-bench → Bug Fixing - swt-bench → Frontend Development - gaia → Information Gathering This ensures category aggregation works even without config file. Tested locally: ✅ Shows real scores (57.767, 54.692, etc.) Co-authored-by: openhands <openhands@all-hands.dev>b4ac443 -
Add debug logging to track data loading on HuggingFace Space
openhands +1: openhands- Add data integrity check in setup_data.py - Add sample data logging in simple_data_loader.py - This will help identify where the zero scores are coming from Co-authored-by: openhands <openhands@all-hands.dev>044cdf4 -
Force rebuild: Trigger HuggingFace Space to fetch latest data from GitHub
openhands +1: openhandsThe Space appears to be using cached data or needs to re-clone the openhands-index-results repository. This commit forces a rebuild to ensure the latest data with real scores is loaded. Co-authored-by: openhands <openhands@all-hands.dev>8be216f -
Fix Categories Attempted calculation to handle missing category columns correctly
openhands- Changed logic to check if category score is not None and not 0.0 - This fixes the issue where missing categories were being counted as attempted - Now correctly shows X/5 based on actual categories with data0718569 -
Remove unused AstaBench category files and update UI to OpenHands categories
openhandsCleanup actions: - Deleted 4 unused AstaBench category files: c_and_e.py, data_analysis.py, e2e.py, literature_understanding.py - Updated ui_components.py tooltips to show OpenHands categories (Bug Fixing, Frontend Development, etc.) instead of AstaBench categories - Changed tooltip from 'four categories' to 'five categories' - Updated leaderboard_transformer.py comment to reflect actual OpenHands field mappings - Category list now: Bug Fixing, Frontend Development, App Creation, Test Generation, Information Gathering This removes ~1KB of dead code and fixes user-facing tooltips to match actual categories.6a0d1cb -
Add Acknowledgements section crediting AstaBench
openhands- Added new Acknowledgements section to About page - Credits AstaBench leaderboard (Allen Institute for AI) for UI design and methodology - Lists key aspects adapted: macro-averaging, visualization, UI structure - Positioned before Citation section737a3f2 -
Fix score calculation to match AstaBench methodology and update categories
openhandsCritical fixes: - Changed overall score calculation from simple average of all benchmarks to macro-average of category averages - This ensures each category contributes equally regardless of benchmark count - Updated Categories Attempted from 4 to 5 categories (Bug Fixing, Frontend Dev, App Creation, Test Gen, Info Gathering) - Updated category list in leaderboard_transformer.py line 281 - Updated tooltip text to reflect 5 categories This matches the AstaBench calculation methodology.e734bf6