Spaces:
Runtime error
Runtime error
Commit History
-
Fix citation to arXiv format and add keep-alive workflow
- Replace @inproceedings with correct @article{levy2024st across README.md, leaderboard_space/README.md, and leaderboard_space/app.py - Add GitHub Actions cron job to ping HF Space every 6h to prevent sleepfd230ed
-
Update HuggingFace URLs from dolev31 to ST-WebAgentBench org
Space, dataset, and data repo moved to the ST-WebAgentBench organization.bb52ba7
-
Update ICLR year from 2025 to 2026 across all sources
b3a385c -
Leaderboard: admin force-add, hardened submission pipeline, and synced task data
c173223 -
Fix 5 submission pipeline issues: sync, IDs, locking, admin targeting
1. Auto-sync deploy script (scripts/deploy_space.py) — copies test.raw.json and computes canonical_hashes.json before deploying, eliminating stale data. Tests catch drift before deploy. 2. Task ID validation uses actual IDs from test.raw.json instead of assuming contiguous range(0, N). Both packages updated. 3. File locking (fcntl.flock) on SUBMISSIONS_FILE writes prevents corruption from concurrent uploads. 4. Admin remove now accepts optional run_id to target a specific submission. When multiple match by agent_id alone, lists them with run_id/date/CuP so admin can pick one. 5. Deploy sync tests (TestDeploySync) verify test.raw.json and canonical_hashes.json are in sync before any deploy.be68cd2 -
Harden submission upload: file size limits, consistent paths, Gradio 6 compat
- Add file size validation (50MB max) and empty file check before JSON parsing - Use __file__-relative paths for all data files (submissions, key_requests, persistence dir) for consistent behavior regardless of CWD - Fix Gradio 6 dropdown updates: use gr.update(choices=...) instead of creating new gr.Dropdown instances (preserves multiselect/label properties) - Update canonical_hashes.json to match current source filesbd7e5b6 -
Fix path resolution and sync canonical test.raw.json for submission validation
- Use __file__-relative paths in schema.py and app.py so validation works regardless of working directory (fixes test imports and Space deployment) - Sync leaderboard_space/data/test.raw.json with canonical stwebagentbench/test.raw.json to fix policy template_id mismatches during submission validation73a5f3a
-
Remove redundant 'Ranked by CuP' label from leaderboard table
The default sort is already indicated by the Sort By dropdown.cc16a38 -
Fix table bottom closure and improve IBM logo quality
- Tables: add overflow:hidden to .table-wrap and 2px bottom border on last row for proper visual closure - Logo: upgrade from blurry 200x200 to crisp 512x512 (~32KB PNG)49b998b -
Fix table bottom closure: add overflow hidden and last-row border
Tables were missing visual bottom closure due to content overflowing the rounded border container. Added overflow:hidden to .table-wrap and a 2px bottom border on the last row.2b1c2c0 -
Embed IBM logo as base64 constant to avoid LFS/Xet issues
HF Spaces don't properly pull LFS objects at runtime, and HF now rejects raw binary pushes (requires Xet). Embedding a compressed 200x200 version (~9KB base64) directly in the code works reliably.5ee0240 -
Remove hardcoded stats: load all config dynamically from test.raw.json
- Include test.raw.json in Space deployment (was excluded by .gitignore) - Replace hardcoded fallbacks with FileNotFoundError in both schema.py files - Replace hardcoded "3 web applications" with len(WEB_APPLICATIONS) - Update README stats to match actual data (375 tasks, 3005 policies)a75cabf -
Make persistence status logs visible in HF Space container logs
Upgrade logger.info → logger.warning for persistence init messages, since HF Spaces default logging level only shows WARNING+.629c2ae -
Fix HF token detection: explicitly read HF_TOKEN from os.environ
HfApi() auto-detection was failing in the Space's Python 3.13 runtime, causing persistence to always be disabled. Now explicitly passes the token from os.environ.get("HF_TOKEN") to HfApi(token=...).853a730 -
Show truncated signing key in admin key request dashboard
Admin can now see the first 16 chars of each user's derived signing key in the Key Request Dashboard table, making it easier to verify and troubleshoot key issues.cfac2c7 -
Replace ad-hoc persistence with CommitScheduler
The previous _persist_file() silently swallowed errors, so data was never actually persisted to the HF dataset repo. Replace with huggingface_hub CommitScheduler which auto-syncs data/ every 2 min. - Remove _persist_file, _restore_data_files, _get_hf_api - Add _init_persistence() with CommitScheduler setup - Download existing data on startup, then schedule auto-commits - Add persistence status banner in admin panel - Add startup health log with record counts - Update SDK version in README from 5.12.0 to 6.6.0a811e8c -
Fix table headers, logo size, and filter row styling
- Override Gradio 6 .header-content white-space:normal with nowrap - Override Gradio 6 thead/tbody table-layout:fixed with auto - Increase IBM logo from 40px to 64px height - Polish filter row with flexbox alignment and spacing8a16b2b -
Fix broken IBM logo and table column rendering
- Base64-encode IBM logo inline to avoid Gradio static file serving issues - Remove table-layout:fixed and width:max-content CSS that broke column layout - Remove column_widths parameter from Dataframe - Increase logo height from 28px to 40px for better visibility4da226e -
Fix logo serving with gr.set_static_paths and table column wrapping
4648178 -
Improve HF token detection for data persistence
2413cf4 -
Improve leaderboard: dynamic stats, per-app table, IBM logo, and UX fixes
- Make task/policy counts, web app count, and tier definitions dynamic from test.raw.json instead of hardcoded values - Add task_metadata with difficulty_tier to CRM tasks 235-294 - Restructure Per-App table to flat layout (Agent|App|CuP|CR|semi-CuP|Gap%|Tasks) - Fix "Ranked by" label to update when sort metric changes - Fix table column rendering with column_widths and white-space CSS - Add IBM logo to leaderboard header - Extend PerAppMetrics schema with semi_CR and semi_CuP fieldse9c0356 -
Add data persistence via private HF dataset repo
Runtime data now survives Space restarts: - _restore_data_files() on startup downloads from dolev31/st-webagentbench-data - _persist_file() uploads after every write (submissions, key requests, audit) - Private dataset repo, owner-only access via HF_TOKENd7ede03 -
Harden admin security: timing-safe auth, rate limiting, sessions, audit
- hmac.compare_digest() for password comparison (prevents timing attacks) - Rate limiting: 5 failed attempts / 5 min → 10 min lockout - Session tokens via gr.State (1-hour TTL, no per-action re-auth) - Input sanitization for key request form (team, institution) - CSV temp files restricted to 0o600 permissions - Append-only admin audit log with vieweraf68c03 -
Add comprehensive admin key request dashboard
Replace plain-text key request log with full dashboard: - Summary statistics (totals, activity, top teams/institutions) - Cumulative timeline chart + institution bar chart - Sortable data table + CSV downloade84798f
-
Redesign leaderboard UI with polished theme, hero header, and consistent styling
- Add custom Gradio theme (blue-indigo palette, Inter/JetBrains Mono fonts) - Add dark gradient hero header with IBM Research logo, ICLR 2025 badge, stats strip - Add ~200 lines of custom CSS: underline tabs, polished tables, form cards, FAQ styling - Consolidate Frontier tab into Leaderboard (removes duplicate Pareto plot) - Rename tabs for cleaner navigation (Safety, Tiers, Per-App, Get Key) - Add Plotly style helpers for consistent chart styling and polished empty states - Add responsive breakpoints for mobilef82bd7d -
Remove Admin tab; embed admin controls in About tab
All admin controls now live inside the "Maintainer Access" accordion in the About tab, hidden behind a password-gated Column. No admin UI is visible to regular users.9762f1e -
Hide admin tab behind password gate; fix import-time env read
- ADMIN_PASSWORD now read dynamically via _get_admin_password() (same pattern as _get_master_key()) to fix stale env var issue - Admin tab hidden by default (visible=False); revealed only after entering correct password via "Maintainer Access" in About tab - Regular users see no admin UI2a6fe08 -
Fix admin password read at import time
Change ADMIN_PASSWORD from a module-level constant to a dynamic _get_admin_password() function that reads the env var at call time. This matches the pattern used for _get_master_key() and ensures the Space picks up the secret even if it was set after the process started.3b44087 -
Add FAQ tab and per-user signing key management to leaderboard
Adds a comprehensive FAQ tab (27 questions across 7 sections) covering setup, signing keys, submission generation, validation errors, metrics, rate limits, and contact guidance — reducing unnecessary support emails. Also includes per-user signing key derivation, key request logging, and admin key request viewer.148457e -
Add 80 modality-challenge tasks and leaderboard submission system
- Add 80 new SuiteCRM tasks (IDs 295-374) testing vision vs. DOM modality gaps: - 40 vision-advantage tasks (V1-V8): aria-hidden labels, CSS colors, canvas, transforms, overlays, emoji navigation, background images - 40 DOM-advantage tasks (D1-D8): invisible elements, tiny text, overlapping elements, off-screen content, dynamic content, truncated text, hidden columns, shadow DOM - Add leaderboard submission system with HMAC signing and 6-layer verification - Add HuggingFace Spaces leaderboard app with validation pipeline - Update README with modality-challenge documentation and 375 total tasks - Add task generator, browser verification script, and test suite (298 tests) - Add SPA wait logic in task.py for Angular content renderingae07f06