Spaces:
Running
Running
Commit History
-
change: paper results win on conflict (official results immutable) Running
Per request, flip the leaderboard merge so the bundled paper results are authoritative: on a (model, adapter) clash the paper row wins, and community submissions from braindecode/contents can only ADD new model/strategy combos, never override an official result.5d5fb3c -
feat: merge real (paper) results with new community results
Previously the board showed HF contents OR the bundled sample_results.json (fallback) — so the first worker write would have hidden the real paper results. Now the leaderboard treats the bundled paper results as an immutable base and OVERLAYS new results from braindecode/contents, deduped by (model, adapter); a freshly evaluated result wins on a same-key clash, every other paper result is preserved. Falls back to base-only when contents is absent. Also alias oeb 'frozen' -> arena 'probe' in results mapping so new linear-probe results align with the paper rows and the frontend filters. Tests: leaderboard merge + frozen->probe alias (22 passing).19eb9db -
fix: results mapping returns [] when a run produced no results
The live in-Space worker test surfaced this: when every experiment fails, oeb's DataFrame has no 'test_balanced_accuracy' column, and the mapper indexed it directly -> KeyError('test_balanced_accuracy'), which masked the real failure. Guard the missing/empty metric column and return [] so the worker reports the clean 'benchmark produced no completed results' error instead. Adds regressions.e42d567 -
feat: run the eval worker inside the Space (CPU, isolated venv)
Make the Space self-contained — it now runs the eval engine itself on CPU, no external machine needed: - Dockerfile: build an isolated /opt/worker-venv (CPU torch + open-eeg-bench), kept OUT of the web app's pinned stack to avoid version conflicts; launch the worker as a background '--watch' loop alongside uvicorn + the frontend, gated by RUN_EVAL_WORKER (set =0 in Space settings to disable without a rebuild). Built before the code COPY (cached across code rebuilds) and non-fatal (a failed worker-deps install never breaks the web app). - run_eval_worker.py: import-light (only huggingface_hub + pure app modules, so it runs in the minimal venv), default --device cpu, add process_queue() and a --watch/--interval service loop. - app/services/__init__.py: stop eagerly importing the heavy submodules so the pure results_mapping can be imported without dragging in datasets/transformers/ aiohttp. All call sites use direct submodule imports, so nothing breaks. Verified: the w…c54d8a2 -
feat: complete the eval engine — real worker + OEB-aware queue reader
The arena's submission loop was open: nothing produced leaderboard results. - run_eval_worker.py is now a complete one-shot worker (run on external GPU/SLURM infra, not the Space): PENDING -> mark RUNNING -> oeb.benchmark() -> map_oeb_results_to_contents -> append to public braindecode/contents (single train.parquet, deduped by model+adapter) -> mark FINISHED/FAILED. Supports --device, --dry-run, --slurm, --infra-folder, --limit. Writes parquet via pandas + upload_file to stay robust across datasets/pyarrow. - _refresh_models_cache now parses the OpenEEGBench request schema (model_name/submitter/benchmark_kwargs) instead of the legacy LLM fields (model/revision/sender), which previously KeyError'd on every entry so submissions never appeared in the queue UI. Legacy fields kept as fallback. - worker-requirements.txt pins open-eeg-bench for the worker env. Verified end-to-end against throwaway HF repos: publish() round-trips through load_dataset(...)['train'] (what the …356d76f -
fix: treat a missing/empty eval queue as empty, not an error
The models-status read path (_refresh_models_cache) raised when braindecode/requests did not exist, surfacing 'Failed to fetch models' in the Evaluation Queues UI. Treat a missing or empty queue dataset as zero pending/evaluating/finished models instead.f00de2f -
fix: self-bootstrap the private eval queue dataset on submit
braindecode/requests did not exist, and submit_model called upload_file without creating it, so a live submission would 404 even on the Space. Create it with exist_ok=True before upload, and keep it PRIVATE because request entries contain the submitter's contact email.6ff21c9 -
feat: OpenEEGBench submission flow (arena runs the eval)
Replace the leftover LLM-leaderboard submission with a first-class OpenEEGBench flow. Users submit a model reference (model_cls + checkpoint + finetuning strategy/head/datasets); the backend validates it, enforces a 5-per-month per-user cap, and queues a request to braindecode/requests. A worker (stub) runs oeb.benchmark() and maps results to braindecode/contents. Backend: - submission_config: OEB enums/defaults (single source of truth) - submissions: pure validation, request building, monthly cap - results_mapping: OEB DataFrame -> contents schema (balanced_accuracy) - model_validation: EEG/OEB weights-source + license checks - OEB-shaped submit endpoint/service; eval worker stub (run_eval_worker.py) Frontend: - OEB submission form (model_cls, weights source, strategies, heads, PEFT modules, datasets, seeds, contact/visibility) with conditional fields - rewritten submission guide (5/month, help -> OpenEEGBench issues, citation) - queue/limit copy aligned; fixed duplicate style key…4db3be0
-
Fill model metadata in sample_results, drop unused fields
Fill #Params (M) with backbone-only param counts (head/final_layer excluded), instantiated from open-eeg-bench backbone configs: BIOT=3.19M, LaBraM=5.82M, CBraMod=4.92M, REVE=69.19M, EEGPT=null (no config in backbones.py). Set Hub License to bsd-3-clause for all entries. Drop embed_dim (dataset-dependent), trainable_params (dataset-dependent), and Weight type (Open LLM Leaderboard artefact, not relevant for EEG PEFT submissions) from the data and from every place that consumed them (transform_data, columnUtils, defaults.js, submission form, ModelService eval-queue path).8d3e1e1
-
Add HF eval-results benchmark registry and API endpoint
- Add canonical EEG_BENCHMARKS registry (8 datasets) in config/benchmarks.py - Add GET /api/benchmarks endpoint for benchmark metadata - Refactor leaderboard transform_data to loop over registry instead of hardcoding - Extract _build_entry_id helper for consistent unique ID generation - Add push_eval_yaml.py script to register benchmarks on HF Hub - Add pyyaml dependency6f5cc9b
-
Fix server 500 error and replace HF logos with eegdash branding
- Add local sample data fallback when braindecode/contents HF dataset is unavailable (backend/app/data/sample_results.json with 20 entries) - Make fastapi-cache2 imports robust with graceful no-op fallback - Replace Hugging Face SVG logo with eegdash brain+EEG wave logo - Generate new favicon PNGs (32px, 256px) from eegdash SVG - Add missing http-proxy-middleware dependency to frontend package.json - Update OG URL to new Space name (OpenEEGBench) - Update AuthContainer to remove HF-specific branding text5b89071 -
Align frontend and backend with adapter_finetuning paper
Replace all LLM-specific references with real EEG benchmark data: - 8 EEG datasets (BCIC-2a, PhysioNet, ISRUC, TUAB, TUEV, CHB-MIT, FACED, SEED-V) - 7 adapter methods (LoRA, IA3, AdaLoRA, DoRA, OFT, Probe, Full Fine-tune) - 7 foundation models (LaBraM, EEGPT, BIOT, BENDR, SignalJEPA, CBraMod, REVE) - Remove official providers, MoE, flagged, merged features - Update params from billions to millions scale - Add real citations and dataset descriptions from the paper5763bf2 -
Migrate from Gradio to React + FastAPI architecture
Replace the old Gradio-based leaderboard with a React frontend and FastAPI backend, adapted from the Open LLM Leaderboard codebase for EEG model evaluation. Backend: FastAPI app with leaderboard, model submission, and voting endpoints. Frontend: React + MUI with leaderboard table, filters, model submission, voting, about page, and citations. Infrastructure: Dockerfile, docker-compose, and env configuration for HF Spaces.ee472e7